# rss_item

guid: https://jvns.ca/blog/2022/02/23/getaddrinfo-is-kind-of-weird/
title: Some things about getaddrinfo that surprised me
url: https://jvns.ca/blog/2022/02/23/getaddrinfo-is-kind-of-weird/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! Here are some things you may or may not have noticed about DNS:</p>

<ul>
<li>when you resolve a DNS name in a Python program, it checks <code>/etc/hosts</code>, but when you use <code>dig</code>, it doesn&rsquo;t.</li>
<li>switching Linux distributions can sometimes change how your DNS works, for example if you use Alpine Linux instead of Ubuntu it can cause problems.</li>
<li>Mac OS has DNS caching, but Linux doesn&rsquo;t necessarily unless you use <code>systemd-resolved</code> or something</li>
</ul>

<p>To understand all of these, we need to learn about a function called
<code>getaddrinfo</code> which is responsible for doing DNS lookups.</p>

<p>There are a bunch of surprising-to-me things about <code>getaddrinfo</code>, and once I
learned about them, it explained a bunch of the confusing DNS behaviour I&rsquo;d
seen in the past.</p>

<h3 id="where-does-getaddrinfo-come-from">where does <code>getaddrinfo</code> come from?</h3>

<p><code>getaddrinfo</code> is part of a library called <code>libc</code> which is the standard C
library. There are at least 3 versions of libc:</p>

<ol>
<li>glibc (GNU libc)</li>
<li>musl libc</li>
<li>the Mac OS version of libc (I don&rsquo;t know if this has a name)</li>
</ol>

<p>There are definitely more (I assume FreeBSD and OpenBSD each have their own
version for example), but those are the 3 I know about.</p>

<p>Each of those have their own version of <code>getaddrinfo</code>.</p>

<h3 id="not-all-programs-use-getaddrinfo-for-dns">not all programs use <code>getaddrinfo</code> for DNS</h3>

<p>The first thing I found surprising is that <code>getaddrinfo</code> is very widely used
but not universally used.</p>

<p>Every program has basically 2 options:</p>

<ol>
<li>use <code>getaddrinfo</code>. I think that Python, Ruby, and Node use <code>getaddrinfo</code>, as well as Go sometimes. Probably many more languages too but I did not have the time to go hunting through every language&rsquo;s DNS library.</li>
<li>use a custom DNS resolver function. Examples of this:

<ul>
<li>dig. I think this is because dig needs more control over the DNS query
than <code>getaddrinfo</code> supports so it implements its own DNS logic.</li>
<li>Go also has a pure-Go DNS resolver if you don&rsquo;t want to use CGo</li>
<li>There&rsquo;s a Ruby gem with a <a href="https://ruby-doc.org/stdlib-2.5.1/libdoc/resolv/rdoc/Resolv.html">custom DNS resolver</a> that you can use to replace <code>getaddrinfo</code>.</li>
<li><code>getaddrinfo</code> doesn&rsquo;t support DNS over HTTPS, so I assume that browsers
that use DoH are not using <code>getaddrinfo</code> for those DNS lookups</li>
<li>probably lots more that I&rsquo;m not aware of</li>
</ul></li>
</ol>

<h3 id="you-ll-sometimes-see-getaddrinfo-in-your-dns-error-messages">you&rsquo;ll sometimes see <code>getaddrinfo</code> in your DNS error messages</h3>

<p>Because <code>getaddrinfo</code> is so widely used, you&rsquo;ll often see it in error messages related to DNS.</p>

<p>For example if I run this Python program which looks up nonexistent domain name:</p>

<pre><code>import requests

requests.get(&quot;http://xyxqqx.com&quot;)
</code></pre>

<p>I get this error message:</p>

<pre><code>Traceback (most recent call last):
  File &quot;/usr/lib/python3.10/site-packages/urllib3/connection.py&quot;, line 174, in _new_conn
    conn = connection.create_connection(
  File &quot;/usr/lib/python3.10/site-packages/urllib3/util/connection.py&quot;, line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File &quot;/usr/lib/python3.10/socket.py&quot;, line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
</code></pre>

<p>I think <code>socket.getaddrinfo</code> is calling libc <code>getaddrinfo</code> somewhere under the
hood, though I did not read all of the source code to check.</p>

<p>Before you learn what <code>getaddrinfo</code> is, it&rsquo;s not at all obvious that
<code>socket.gaierror: [Errno -2] Name or service not known</code> means &ldquo;that domain
doesn&rsquo;t exist&rdquo;. It doesn&rsquo;t even say the words &ldquo;DNS&rdquo; or &ldquo;domain&rdquo; in it
anywhere!</p>

<h3 id="getaddrinfo-on-mac-doesn-t-use-etc-resolv-conf"><code>getaddrinfo</code> on Mac doesn&rsquo;t use <code>/etc/resolv.conf</code></h3>

<p>I used to use a Mac for work, and I always felt vaguely unsettled by DNS on
Mac. I could tell that <strong>something</strong> was different from how it worked on my
Linux machine, but I couldn&rsquo;t figure out what it was.</p>

<p>I still don&rsquo;t totally understand this and it&rsquo;s hard for me to investigate
because I don&rsquo;t currently have access to a Mac but here&rsquo;s what I&rsquo;ve gathered
so far.</p>

<p>On Linux systems, <code>getaddrinfo</code> decides which DNS resolver to talk to using a
file called <code>/etc/resolv.conf</code>. <small>(there&rsquo;s apparently some additional
complexity with <code>/etc/nsswitch.conf</code> but I have never looked at
<code>/etc/nsswitch.conf</code> so I&rsquo;m going to ignore it).</small></p>

<p>For example, this is the contents of my <code>/etc/resolv.conf</code> right now:</p>

<pre><code># Generated by NetworkManager
nameserver 192.168.1.1
nameserver fd13:d987:748a::1
</code></pre>

<p>This means that to make DNS queries, <code>getaddrinfo</code> makes a request to
<code>192.168.1.1</code> on port 53. That&rsquo;s my router&rsquo;s DNS resolver.</p>

<p>I assumed this was <code>getaddrinfo</code> on Mac also just used <code>/etc/resolv.conf</code> but I was wrong.
Instead, <code>getaddrinfo</code> makes a request to a program called <code>mDNSResponder</code>
which is a Mac thing.</p>

<p>I don&rsquo;t know much about <code>mDNSResponder</code> except that it does DNS caching and
that apparently you can clear the cache with <code>dscacheutil</code>. This explains one
of the mysteries at the beginning of the post &ndash; why Macs have DNS caching and
Linux machines don&rsquo;t always.</p>

<h3 id="musl-libc-getaddrinfo-is-different-from-glibc-s-version">musl libc <code>getaddrinfo</code> is different from glibc&rsquo;s version</h3>

<p>You might think ok, Mac OS <code>getaddrinfo</code> is different, but the two versions of
<code>getaddrinfo</code> in glibc and musl libc must be mostly the same, right?</p>

<p>But they have some pretty significant differences. The main difference I know
about is that musl libc does not support TCP DNS. I couldn&rsquo;t find anything in
the documentation about it but it&rsquo;s mentioned in <a href="https://twitter.com/RichFelker/status/994667677112156161">this tweet</a>)</p>

<p>I talked a bit more about this TCP DNS thing in <a href="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/">ways DNS can break</a>.</p>

<p>Some more differences:</p>

<ul>
<li>the way search domains (in <code>/etc/resolv.conf</code>) are handled is slightly different (<a href="https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS">discussed here</a>)</li>
<li>this post mentions that <a href="https://utcc.utoronto.ca/~cks/space/blog/linux/GlibcAndLinuxAPI">musl doesn&rsquo;t support nsswitch.conf</a>.
I have never used nsswitch.conf and I&rsquo;m not sure why it&rsquo;s useful but I think
there are reasons I don&rsquo;t know about.</li>
</ul>

<h3 id="more-weird-things-nscd">more weird things: nscd?</h3>

<p>When looking up getaddrinfo I also found this interesting <a href="https://jameshfisher.com/2018/02/03/what-does-getaddrinfo-do/">post about getaddrinfo from James Fisher</a> that
straces glibc <code>getaddrinfo</code> and discovers that apparently calls some
program called <code>nscd</code> which is supposed to do DNS caching. That blog post
describes nscd as &ldquo;unstable&rdquo; and &ldquo;badly designed&rdquo; and it&rsquo;s not clear to me how
widely used it is.</p>

<p>I don&rsquo;t know anything about nscd but I checked and apparently it&rsquo;s on my
computer. I tried it out and this is what happened:</p>

<pre><code>$ nscd 
child exited with status 4
</code></pre>

<p>My impression is that people who want to do DNS caching on Linux are more
likely to use a DNS forwarder like <code>dnsmasq</code> or <code>systemd-resolved</code> instead of
something like <code>nscd</code> &ndash; that&rsquo;s what I&rsquo;ve seen in the past.</p>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>When I first learned about all of this I found it really surprising that such
a widely used library function has such different behaviour on different
platforms.</p>

<p>I mean, it makes sense that the people who built Mac OS would want to handle
DNS caching in a different way than it&rsquo;s handled on Linux, so it&rsquo;s reasonable
that they implemented <code>getaddrinfo</code> differently.  And it makes sense that some
programs choose not to use <code>getaddrinfo</code> to make DNS queries.</p>

<p>But it definitely makes DNS a bit more difficult to reason about.</p>


guid: https://jvns.ca/blog/2022/03/08/tiny-programs/
title: Some tiny personal programs I've written
url: https://jvns.ca/blog/2022/03/08/tiny-programs/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>I was talking to a friend last summer about what resources might be helpful for
folks learning to program. My friend said they thought some people might
benefit from a list of small and fun programming projects &ndash; the kind of thing
you can do in an evening or weekend.</p>

<p>So let&rsquo;s talk about that! I like to write small programs that have some
marginal utility in my life. Kind of like this:</p>

<ul>
<li>ah! A minor problem in my life!</li>
<li>I know, I bet I can solve this problem with CODE. YAY.</li>
<li><em>4 hours of happy programming ensues</em></li>
</ul>

<p>This isn&rsquo;t always the most <em>practical</em> (many of the problems I&rsquo;ve
solved with programming could have been solved in less time in other ways), but
as long as your goal is actually to have fun programming and your programs
don&rsquo;t hurt anyone else, I think this is a great approach :)</p>

<p>So here are a few examples of small personal programming projects
I&rsquo;ve done. I&rsquo;m not going to talk about &ldquo;learning projects&rdquo; where my goal was
to learn something specific because I&rsquo;ve already written a billion blog posts
about that.</p>

<p>These are more about just doing something fun with no specific learning goal.</p>

<h3 id="a-theatre-festival-didn-t-have-a-calendar">a theatre festival didn&rsquo;t have a calendar</h3>

<p>The local Fringe Festival had a bunch of shows, but there was no place I could
see a calendar all one one page. So I wrote a Python script to scrape their
website and generate a calendar. Here&rsquo;s <a href="https://gist.github.com/jvns/a552895b6f9b523276e88d7e7506ee86">the code</a> and <a href="https://jvns.ca/fringe-2021.html">the output</a>.</p>

<h3 id="printing-out-covers-for-tiny-books">printing out covers for tiny books</h3>

<p>I saw a <a href="https://www.tiktok.com/@stevieandjohn/video/7063929340836334895">TikTok video</a> recently where someone made miniature physical versions of the
ebooks they read. I decided to try it out, so I needed to print tiny versions
of a bunch of book covers. I could have resized all of them manually, but I
decided to do it with programming instead.</p>

<p>So I wrote a little bit of HTML and CSS
(<a href="https://gist.github.com/jvns/de0671d09f5752d36cbe6082d4fb2af2">tinybooks.html</a>), converted it to a PDF, and printed it out.</p>

<h3 id="getting-my-scanner-to-work-better">getting my scanner to work better</h3>

<p>This is barely &ldquo;programming&rdquo;, but I needed to scan a bunch of documents for a
family member, and I didn&rsquo;t like the available software. So I wrote a <a href="https://gist.github.com/jvns/b5651fd6272eddcd935e5e30874a935f">tiny shell script wrapper for <code>scanimage</code></a> to
make the process simpler. This one actually helped me a lot and I still use it when
scanning.</p>

<h3 id="getting-a-vaccine-appointment">getting a vaccine appointment</h3>

<p>When the second COVID vaccine doses opened up, all of the slots were full. It
turned out that the website&rsquo;s backend had an API, so I wrote a script to poll
the API every 60 seconds or so and watch for cancellations and notify me so
that I could get an earlier appointment.</p>

<p>This didn&rsquo;t turn out to be necessary (more appointments opened up pretty soon
anyway and there were enough for everyone), but it was fun.</p>

<p>In general I try to be careful when using APIs like this in a way the
developers didn&rsquo;t intend to avoid overloading the site.</p>

<h3 id="looking-at-housing-market-data">looking at housing market data</h3>

<p>We were thinking of buying a condo a few years ago and I was mad that I
couldn&rsquo;t get any information about historical prices, so I wrote an iPython
notebook that queried the API of a local real estate website to scrape some
information and calculate some statistics like price per square foot over time.</p>

<p>I don&rsquo;t think this actually helped us at all with buying a condo but it was fun.</p>

<p>(&ldquo;using the API of local services&rdquo; seems to be an ongoing theme, one of my
favourite things is to use secret undocumented APIs where you need to copy your
cookies out of the browser to get access to them)</p>

<h3 id="crossword-business-cards">crossword business cards</h3>

<p>in 2013, I thought it might be fun to have a business card that was a crossword
with some of my interests. So I wrote general software to <a href="https://github.com/jvns/svg_crossworder">generate crosswords from a text file</a>. I&rsquo;m pretty sure
never printed the business cards but it was fun to write.</p>

<h3 id="generating-envelopes">generating envelopes</h3>

<p>I was mailing some zines a while ago, and I decided I wanted to print custom
labels on every envelope &ndash; sort of a &ldquo;mail merge&rdquo; situation. So I wrote a
Python program to go through all of the mailing address and generate some HTML
and CSS. Then I turned the HTML/CSS into a PDF and printed the envelopes. This
worked great.</p>

<h3 id="investigating-dice-rolling-patterns">investigating dice rolling patterns</h3>

<p>A friend showed me a dice rolling game where you roll a bunch of dice and add
up the values. I mentioned that if you roll enough dice and add up all the
values, at some point it gets a lot less &ldquo;random&rdquo;.</p>

<p>But then I wanted to see exactly how much less random it gets. So I wrote a
tiny program to roll 2500 dice and add up the resulting sums a bunch of times
to see how it works. (presumably you could calculate the same thing with math,
but it&rsquo;s easier with code)</p>

<p>This was so little code I&rsquo;ll just inline it here. (it&rsquo;s Python). Here&rsquo;s [the output](<a href="https://gist.github.com/jvns/e4a35ca2bad90c1a0fcaf578a803b456">https://gist.github.com/jvns/e4a35ca2bad90c1a0fcaf578a803b456</a></p>

<pre><code>import random

def roll():
    return sum(random.randint(1, 6) for i in range(2500))

while True:
    print(roll())
</code></pre>

<h3 id="getting-drawings-into-the-notability-app">getting drawings into the Notability app</h3>

<p>I was using an app called Squid to do drawing, and I was switching to
Notability and wanted to get my old drawings into Notability. So I <a href="https://jvns.ca/blog/2018/03/31/reverse-engineering-notability-format/">reverse engineered the Notability file format</a>.</p>

<p>I don&rsquo;t think this was ultimately that useful (I ultimately ended up switching
to a different drawing app which had a real SVG import), but I had fun.</p>

<h3 id="turning-off-retweets">turning off retweets</h3>

<p>This is a slightly less tiny project (it took more than one day), but I decided I didn&rsquo;t want to see retweets
on Twitter anymore so I
wrote a <a href="https://turn-off-retweets.glitch.me/">small website</a> so I could turn off retweets.</p>

<h3 id="i-really-love-tiny-projects">I really love tiny projects</h3>

<p>All of these examples are more recent, but I think that when I was starting to learn to
program tiny low-stakes projects like this really helped me. I love that</p>

<ul>
<li>they&rsquo;re just for me (if it goes wrong, it doesn&rsquo;t matter!)</li>
<li>I can finish them in an evening or weekend (it&rsquo;s not a Huge Giant Thing hanging over my head)</li>
<li>if it works, there&rsquo;s some tangible output in my life (like some envelopes or miniature books or a schedule a business card or a better Twitter experience)</li>
</ul>


guid: https://jvns.ca/blog/2022/03/10/how-to-use-undocumented-web-apis/
title: How to use undocumented web APIs
url: https://jvns.ca/blog/2022/03/10/how-to-use-undocumented-web-apis/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! A couple of days I wrote about <a href="https://jvns.ca/blog/2022/03/08/tiny-programs/">tiny personal programs</a>, and I mentioned that
it can be fun to use &ldquo;secret&rdquo; undocumented APIs where you need to copy your
cookies out of the browser to get access to them.</p>

<p>A couple of people asked how to do this, so I wanted to explain how because
it&rsquo;s pretty straightforward. We&rsquo;ll also talk a tiny bit about what can go
wrong, ethical issues, and how this applies to your undocumented APIs.</p>

<p>As an example, let&rsquo;s use Google Hangouts. I&rsquo;m picking this not because it&rsquo;s the
most useful example (I think there&rsquo;s an official API which would be much more
practical to use), but because many sites where this is actually useful are
smaller sites that are more vulnerable to abuse. So we&rsquo;re just going to use
Google Hangouts because I&rsquo;m 100% sure that the Google Hangouts backend is
designed to be resilient to this kind of poking around.</p>

<p>Let&rsquo;s get started!</p>

<h3 id="step-1-look-in-developer-tools-for-a-promising-json-response">step 1: look in developer tools for a promising JSON response</h3>

<p>I start out by going to <a href="https://hangouts.google.com">https://hangouts.google.com</a>, opening the network tab in
Firefox developer tools and looking for JSON responses. You can use Chrome developer tools too.</p>

<p>Here&rsquo;s what that looks like</p>

<p><img src="https://jvns.ca/images/network-tab.png"></p>

<p>The request is a good candidate if it says &ldquo;json&rdquo; in the &ldquo;Type&rdquo; column&rdquo;</p>

<p>I had to look around for a while until I found something interesting, but
eventually I found a &ldquo;people&rdquo; endpoint that seems to return information about
my contacts. Sounds fun, let&rsquo;s take a look at that.</p>

<h3 id="step-2-copy-as-curl">step 2: copy as cURL</h3>

<p>Next, I right click on the request I&rsquo;m interested in, and click &ldquo;Copy&rdquo; -&gt; &ldquo;Copy as cURL&rdquo;.</p>

<p>Then I paste the <code>curl</code> command in my terminal and run it. Here&rsquo;s what happens.</p>

<pre><code>$ curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED' -X POST ........ (a bunch of headers removed)
Warning: Binary output can mess up your terminal. Use &quot;--output -&quot; to tell 
Warning: curl to output it to your terminal anyway, or consider &quot;--output 
Warning: &lt;FILE&gt;&quot; to save to a file.
</code></pre>

<p>You might be thinking &ndash; that&rsquo;s weird, what&rsquo;s this &ldquo;binary output can mess up
your terminal&rdquo; error?  That&rsquo;s because by default, browsers send an
<code>Accept-Encoding: gzip, deflate</code> header to the server, to get compressed
output.</p>

<p>We could decompress it by piping the output to <code>gunzip</code>, but I find it simpler
to just not send that header. So let&rsquo;s remove some irrelevant headers.</p>

<h3 id="step-3-remove-irrelevant-headers">step 3: remove irrelevant headers</h3>

<p>Here&rsquo;s the full <code>curl</code> command line that I got from the browser. There&rsquo;s a lot here!
I start out by splitting up the request with backslashes (<code>\</code>) so that each header is on a different line to make it easier to work with:</p>

<pre><code>curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED' \
-X POST \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0' \
-H 'Accept: */*' \
-H 'Accept-Language: en' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'X-HTTP-Method-Override: GET' \
-H 'Authorization: SAPISIDHASH REDACTED' \
-H 'Cookie: REDACTED'
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'X-Goog-AuthUser: 0' \
-H 'Origin: https://hangouts.google.com' \
-H 'Connection: keep-alive' \
-H 'Referer: https://hangouts.google.com/' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-site' \
-H 'Sec-GPC: 1' \
-H 'DNT: 1' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'TE: trailers' \
--data-raw 'personId=101777723309&amp;personId=1175339043204&amp;personId=1115266537043&amp;personId=116731406166&amp;extensionSet.extensionNames=HANGOUTS_ADDITIONAL_DATA&amp;extensionSet.extensionNames=HANGOUTS_OFF_NETWORK_GAIA_GET&amp;extensionSet.extensionNames=HANGOUTS_PHONE_DATA&amp;includedProfileStates=ADMIN_BLOCKED&amp;includedProfileStates=DELETED&amp;includedProfileStates=PRIVATE_PROFILE&amp;mergedPersonSourceOptions.includeAffinity=CHAT_AUTOCOMPLETE&amp;coreIdParams.useRealtimeNotificationExpandedAcls=true&amp;requestMask.includeField.paths=person.email&amp;requestMask.includeField.paths=person.gender&amp;requestMask.includeField.paths=person.in_app_reachability&amp;requestMask.includeField.paths=person.metadata&amp;requestMask.includeField.paths=person.name&amp;requestMask.includeField.paths=person.phone&amp;requestMask.includeField.paths=person.photo&amp;requestMask.includeField.paths=person.read_only_profile_info&amp;requestMask.includeField.paths=person.organization&amp;requestMask.includeField.paths=person.location&amp;requestMask.includeField.paths=person.cover_photo&amp;requestMask.includeContainer=PROFILE&amp;requestMask.includeContainer=DOMAIN_PROFILE&amp;requestMask.includeContainer=CONTACT&amp;key=REDACTED'
</code></pre>

<p>This can seem like an overwhelming amount of stuff at first, but you don&rsquo;t need
to think about what any of it means at this stage. You just need to delete
irrelevant lines.</p>

<p>I usually just figure out which headers I can delete with trial and error &ndash; I
keep removing headers until the request starts failing. In general you probably
don&rsquo;t need <code>Accept*</code>, <code>Referer</code>, <code>Sec-*</code>, <code>DNT</code>, <code>User-Agent</code>, and caching
headers though.</p>

<p>In this example, I was able to cut the request down to this:</p>

<pre><code>curl 'https://people-pa.clients6.google.com/v2/people/?key=REDACTED' \
-X POST \
-H 'Authorization: SAPISIDHASH REDACTED' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Origin: https://hangouts.google.com' \
-H 'Cookie: REDACTED'\
--data-raw 'personId=101777723309&amp;personId=1175339043204&amp;personId=1115266537043&amp;personId=116731406166&amp;extensionSet.extensionNames=HANGOUTS_ADDITIONAL_DATA&amp;extensionSet.extensionNames=HANGOUTS_OFF_NETWORK_GAIA_GET&amp;extensionSet.extensionNames=HANGOUTS_PHONE_DATA&amp;includedProfileStates=ADMIN_BLOCKED&amp;includedProfileStates=DELETED&amp;includedProfileStates=PRIVATE_PROFILE&amp;mergedPersonSourceOptions.includeAffinity=CHAT_AUTOCOMPLETE&amp;coreIdParams.useRealtimeNotificationExpandedAcls=true&amp;requestMask.includeField.paths=person.email&amp;requestMask.includeField.paths=person.gender&amp;requestMask.includeField.paths=person.in_app_reachability&amp;requestMask.includeField.paths=person.metadata&amp;requestMask.includeField.paths=person.name&amp;requestMask.includeField.paths=person.phone&amp;requestMask.includeField.paths=person.photo&amp;requestMask.includeField.paths=person.read_only_profile_info&amp;requestMask.includeField.paths=person.organization&amp;requestMask.includeField.paths=person.location&amp;requestMask.includeField.paths=person.cover_photo&amp;requestMask.includeContainer=PROFILE&amp;requestMask.includeContainer=DOMAIN_PROFILE&amp;requestMask.includeContainer=CONTACT&amp;key=REDACTED'
</code></pre>

<p>So I just need 4 headers: <code>Authorization</code>, <code>Content-Type</code>, <code>Origin</code>, and <code>Cookie</code>. That&rsquo;s a lot more manageable.</p>

<h3 id="step-4-translate-it-into-python">step 4: translate it into Python</h3>

<p>Now that we know what headers we need, we can translate our <code>curl</code> command into a Python program!
This part is also a pretty mechanical process, the goal is just to send exactly the same data with Python as we were with curl.</p>

<p>Here&rsquo;s what that looks like. This is exactly the same as the previous <code>curl</code>
command, but using Python&rsquo;s <code>requests</code>. I also broke up the very long request body
string into an array of tuples to make it easier to work with
programmmatically.</p>

<pre><code>import requests
import urllib

data = [
    ('personId','101777723'), # I redacted these IDs a bit too
    ('personId','117533904'),
    ('personId','111526653'),
    ('personId','116731406'),
    ('extensionSet.extensionNames','HANGOUTS_ADDITIONAL_DATA'),
    ('extensionSet.extensionNames','HANGOUTS_OFF_NETWORK_GAIA_GET'),
    ('extensionSet.extensionNames','HANGOUTS_PHONE_DATA'),
    ('includedProfileStates','ADMIN_BLOCKED'),
    ('includedProfileStates','DELETED'),
    ('includedProfileStates','PRIVATE_PROFILE'),
    ('mergedPersonSourceOptions.includeAffinity','CHAT_AUTOCOMPLETE'),
    ('coreIdParams.useRealtimeNotificationExpandedAcls','true'),
    ('requestMask.includeField.paths','person.email'),
    ('requestMask.includeField.paths','person.gender'),
    ('requestMask.includeField.paths','person.in_app_reachability'),
    ('requestMask.includeField.paths','person.metadata'),
    ('requestMask.includeField.paths','person.name'),
    ('requestMask.includeField.paths','person.phone'),
    ('requestMask.includeField.paths','person.photo'),
    ('requestMask.includeField.paths','person.read_only_profile_info'),
    ('requestMask.includeField.paths','person.organization'),
    ('requestMask.includeField.paths','person.location'),
    ('requestMask.includeField.paths','person.cover_photo'),
    ('requestMask.includeContainer','PROFILE'),
    ('requestMask.includeContainer','DOMAIN_PROFILE'),
    ('requestMask.includeContainer','CONTACT'),
    ('key','REDACTED')
]
response = requests.post('https://people-pa.clients6.google.com/v2/people/?key=REDACTED',
    headers={
        'X-HTTP-Method-Override': 'GET',
        'Authorization': 'SAPISIDHASH REDACTED',
        'Content-Type': 'application/x-www-form-urlencoded',
        'Origin': 'https://hangouts.google.com',
        'Cookie': 'REDACTED',
    },
    data=urllib.parse.urlencode(data),
)

print(response.text)
</code></pre>

<p>I ran this program and it works &ndash; it prints out a bunch of JSON! Hooray!</p>

<p>You&rsquo;ll notice that I replaced a bunch of things with <code>REDACTED</code>, that&rsquo;s because
if I included those values you could access the Google Hangouts API for my
account which would be no good.</p>

<h3 id="and-we-re-done">and we&rsquo;re done!</h3>

<p>Now I can modify the Python program to do whatever I want, like passing
different parameters or parsing the output.</p>

<p>I&rsquo;m not going to do anything interesting with it because I&rsquo;m not actually
interested in using this API at all, I just wanted to show what the process looks like.</p>

<p>But we get back a bunch of JSON that you could definitely do something with.</p>

<h3 id="curlconverter-looks-great">curlconverter looks great</h3>

<p>Someone commented that you can translate curl to Python (and a bunch of other
languages!) automatically with <a href="https://curlconverter.com/">https://curlconverter.com/</a> which looks amazing
&ndash; I&rsquo;ve always done it manually.  I tried it out on this example and it seems
to work great.</p>

<h3 id="figuring-out-how-the-api-works-is-nontrivial">figuring out how the API works is nontrivial</h3>

<p>I don&rsquo;t want to undersell how difficult it can be to figure out how an unknown
API works &ndash; it&rsquo;s not obvious! I have no idea what a lot of the parameters to
this Google Hangouts API do!</p>

<p>But a lot of the time there are some parameters that seem pretty straightforward,
like <code>requestMask.includeField.paths=person.email</code> probably means &ldquo;include each
person&rsquo;s email address&rdquo;. So I try to focus on the parameters I <em>do</em> understand
more than the ones I <em>don&rsquo;t</em> understand.</p>

<h3 id="this-always-works-in-theory">this always works (in theory)</h3>

<p>Some of you might be wondering &ndash; can you always do this?</p>

<p>The answer is sort of yes &ndash; browsers aren&rsquo;t magic! All the information
browsers send to your backend is just HTTP requests. So if I copy all of the
HTTP headers that my browser is sending, I think there&rsquo;s literally no way for
the backend to tell that the request <em>isn&rsquo;t</em> sent by my browser and is actually
being sent by a random Python program.</p>

<p>Of course, we removed a bunch of the headers the browser sent so theoretically
the backend <em>could</em> tell, but usually they won&rsquo;t check.</p>

<p>There are some caveats though &ndash; for example a lot of Google services have
backends that communicate with the frontend in a totally inscrutable (to me)
way, so even though in theory you could mimic what they&rsquo;re doing, in practice
it might be almost impossible. And bigger APIs that encounter more abuse
will have more protections.</p>

<p>Now that we&rsquo;ve seen how to use undocumented APIs like this, let&rsquo;s talk about
some things that can go wrong.</p>

<h3 id="problem-1-expiring-session-cookies">problem 1: expiring session cookies</h3>

<p>One big problem here is that I&rsquo;m using my Google session cookie for
authentication, so this script will stop working whenever my browser session
expires.</p>

<p>That means that this approach wouldn&rsquo;t work for a long running program (I&rsquo;d
want to use a real API), but if I just need to quickly grab a little bit of data as a
1-time thing, it can work great!</p>

<h3 id="problem-2-abuse">problem 2: abuse</h3>

<p>If I&rsquo;m using a small website, there&rsquo;s a chance that my little Python script
could take down their service because it&rsquo;s doing way more requests than they&rsquo;re
able to handle. So when I&rsquo;m doing this I try to be respectful and not make too
many requests too quickly.</p>

<p>This is especially important because a lot of sites which don&rsquo;t have official
APIs are smaller sites with less resources.</p>

<p>In this example obviously this isn&rsquo;t a problem &ndash; I think I made 20 requests
total to the Google Hangouts backend while writing this blog post, which they
can definitely handle.</p>

<p>Also if you&rsquo;re using your account credentials to access the API in a excessive
way and you cause problems, you might (very reasonably) get your account
suspended.</p>

<p>I also stick to downloading data that&rsquo;s either mine or that&rsquo;s intended to be
publicly accessible &ndash; I&rsquo;m not searching for vulnerabilities.</p>

<h3 id="remember-that-anyone-can-use-your-undocumented-apis">remember that anyone can use your undocumented APIs</h3>

<p>I think the most important thing to know about this isn&rsquo;t actually how to use <em>other
people&rsquo;s</em> undocumented APIs. It&rsquo;s fun to do, but it has a lot
of limitations and I don&rsquo;t actually do it that often.</p>

<p>It&rsquo;s much more important to understand that anyone can do this to <em>your</em>
backend API!  Everyone has developer tools and the network tab, and it&rsquo;s pretty
easy to see which parameters you&rsquo;re passing to the backend and to change them.</p>

<p>So if anyone can just change some parameters to get another user&rsquo;s information,
that&rsquo;s no good. I think most developers building publicly availble APIs know
this, but I&rsquo;m mentioning it because everyone needs to learn it for the first
time at some point :)</p>


guid: https://jvns.ca/blog/2022/03/13/celebrate-tiny-learning-milestones/
title: Celebrate tiny learning milestones
url: https://jvns.ca/blog/2022/03/13/celebrate-tiny-learning-milestones/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! Today I want to talk about &ndash; how do you know you&rsquo;re getting better at programming?</p>

<p>One obvious approach is:</p>

<ol>
<li>make goals</li>
<li>periodically check if you achieved those goals</li>
<li>if you did, celebrate</li>
</ol>

<h3 id="i-kind-of-hate-goals">I kind of hate goals</h3>

<p>Goals can be useful, but a lot of the time I actually find them stressful and
not that helpful. For example, here are a few goals I wrote down for myself 9
years ago:</p>

<ul>
<li>write a nontrivial amount of C code that works and is used by actual people</li>
<li>contribute to an open source project in C</li>
<li>learn C++</li>
</ul>

<p>9 years later, I have done 0 of those things. With the &ldquo;goal&rdquo; framing, it&rsquo;s to
think of this as a bad thing! Like, I wanted to learn C++ and I didn&rsquo;t! I still
can&rsquo;t write C comfortably! Oh no! I guess I failed!</p>

<p>I find this framing depressing and unhelpful. In reality, I didn&rsquo;t have any real
reason to learn C++ then and I still don&rsquo;t now. So it makes sense that I
haven&rsquo;t learned it.</p>

<p>Instead of goals, I like to think about <strong>tiny milestones</strong>.</p>

<h3 id="what-s-a-milestone">what&rsquo;s a milestone?</h3>

<p>Usually when we talk about milestones we mean something big that only happens every few years, like &ldquo;I graduated from university&rdquo;.</p>

<p>But in this post I want to talk about milestones in the sense of its etymology
&ndash; <strong>stones</strong> placed every <strong>mile</strong> on a highway, so that you can track your
progress along a path.</p>

<p>These happen much more often &ndash; maybe you used a new tool for the first time,
or you fixed a new type of bug that you&rsquo;ve never seen before, or you learned
about a new concept!</p>

<h3 id="a-few-of-my-tiny-c-milestones">a few of my tiny C milestones</h3>

<p>Here are a few examples of tiny milestones from the last 9 years that are
spiritually related to my original &ldquo;learn C/C++&rdquo; goals.</p>

<p>I&rsquo;m pretty sure that each of these individually took less than a week, though
all together they took many years and a lot of them would have been impossible
for me at the beginning.</p>

<ul>
<li>wrote a tiny Linux kernel module that does almost nothing</li>
<li>learned about <code>strace</code></li>
<li>wrote a very basic shell in C with a friend</li>
<li>learned how ELF binaries are organized (symbols, sections, etc)</li>
<li>learned how to use <code>gdb</code> to inspect a C program&rsquo;s memory</li>
<li>learned a little about how how <code>gdb</code> actually works internally (using DWARF)</li>
<li>learned the diffence between static and dynamic linking</li>
<li>learned how to look at how a program is linked with <code>ldd</code> or <code>file</code></li>
<li>(years later) debugged a problem that was caused by dynamic linking</li>
<li>implemented a buffer overflow exploit using <code>gdb</code> and <code>strace</code> (for a CTF)</li>
<li>got a core dump for a crashing C++ program and managed to get a stack trace out of it</li>
<li>learned about the RAII pattern (though in Rust, not C++)</li>
<li>learned what a few basic x86 assembly instructions mean (<code>mov</code>, etc)</li>
<li>pair programmed with a friend who already knew x86 assembly on implementing one Advent of Code problem (Day 1) in x86 assembly</li>
<li>in general I&rsquo;m comfortable writing very basic C programs as long as they
don&rsquo;t have to do anything fancy like &ldquo;memory management&rdquo;</li>
</ul>

<p>And there were even some major milestones, like I wrote a <a href="https://github.com/rbspy/rbspy">Ruby profiler</a> in Rust in 2018.</p>

<p>When I think about it this way, I feel really good about my skills! I&rsquo;ve
learned all kinds of stuff related to systems programming, it just happened in
a different way than I originally expected.</p>

<h3 id="fixing-a-bug-can-be-a-milestone">fixing a bug can be a milestone</h3>

<p>Every time I solve a bug that I couldn&rsquo;t have solved before, I think of it as a
tiny milestone. For example, I&rsquo;ve been trying to get better at CSS. One big
part of that for me is diving deeper into CSS bugs I encounter instead of
giving up.</p>

<p>Last year, I was having a problem with a flexbox. It was something that I
vaguely felt had happened to me before but that I&rsquo;d never been able to fix, and
it made me feel like I didn&rsquo;t understand how flexbox worked.</p>

<p>But instead of just finding a workaround, I decided to try to understand what was actually happening. And I
ended up finding a blog post that explained what was happening &ndash; <a href="https://weblog.west-wind.com/posts/2016/feb/15/flexbox-containers-pre-tags-and-managing-overflow">Flexbox Containers, PRE tags and managing Overflow</a>.
And that was really the cause of my bug!</p>

<h3 id="changing-goals-isn-t-a-bad-thing">changing goals isn&rsquo;t a bad thing</h3>

<p>The reason I still haven&rsquo;t learned C isn&rsquo;t that I suck or that C is impossible
to learn. It&rsquo;s just that learning how to write C well was never actually
something I had a real reason to do.</p>

<p>Instead I learned Rust and Go and <code>strace</code> and <code>gdb</code> and about C structs and
symbols and the call stack and the heap and lots of other things. (as an aside, I loved this paper <a href="https://www.humprog.org/~stephen/research/papers/kell17some-preprint.pdf">Some were meant for C</a>
about how why C is still so important)</p>

<p>And that worked great! So I think it&rsquo;s much more healthy to be flexible about
your goals and to celebrate the milestones you do end up getting to instead of
feeling bad about goals that you &ldquo;failed&rdquo; at.</p>

<h3 id="you-can-learn-a-lot-by-accident">you can learn a lot by &ldquo;accident&rdquo;</h3>

<p>Most of my tiny milestones came up naturally because I had a project I wanted
to do or a bug I needed to solve. So I didn&rsquo;t need to explicitly plan for them,
they just kind of happened along the way because I kept doing projects that
challenged me.</p>

<h3 id="celebrate-your-tiny-milestones">celebrate your tiny milestones</h3>

<p>It&rsquo;s really helpful for me to <strong>celebrate</strong> tiny milestones like this. I celebrate
a lot by writing blog posts &ndash; I wrote the above list mostly by looking at my
list of old blog posts for things I&rsquo;d written about related to C.</p>

<p>If you don&rsquo;t blog (it&rsquo;s definitely not for everyone!), it can be helpful to write down
this kind of thing in your <a href="https://jvns.ca/blog/brag-documents/">brag document</a> instead.</p>

<p>But I do think it&rsquo;s important to celebrate these milestones <em>somewhere</em>. It
gives me a real sense that I&rsquo;m making progress and it helps me stay motivated
to keep learning about the thing.</p>


guid: https://jvns.ca/blog/2022/03/23/a-toy-version-of-tls/
title: Implementing a toy version of TLS 1.3
url: https://jvns.ca/blog/2022/03/23/a-toy-version-of-tls/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! Recently I&rsquo;ve been thinking about how I find it fun to learn computer
networking by implementing working versions of real network protocols.</p>

<p>And it made me wonder &ndash; I&rsquo;ve implemented toy versions of
<a href="https://jvns.ca/blog/2013/10/31/day-20-scapy-and-traceroute/">traceroute</a>, <a href="https://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-tcp-stack-in-python/">TCP</a> and <a href="https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/">DNS</a>.
What about TLS? Could I implement a toy version of that to learn more about how it works?</p>

<p>I asked on Twitter if this would be hard, got <a href="https://twitter.com/Lukasaoz/status/1505593360521777157">some encouragement and pointers for where to start</a>, so I decided to go for it.</p>

<p>This was really fun and I learned a little more about how involved real
cryptography is &ndash; thanks to <a href="https://cryptopals.com/">cryptopals</a>, I already 100% believed that I should not invent my own
crypto implementations, and seeing how the crypto in TLS 1.3 works gave me even more of
an appreciation for why I shouldn&rsquo;t :)</p>

<p>As a warning: I am really not a cryptography person, I will probably say some
incorrect things about cryptography in this post and I absolutely do not know
the history of past TLS vulnerabilities that informed TLS 1.3&rsquo;s design.</p>

<p>All of that said, let&rsquo;s go implement some cryptography! All of my hacky code is <a href="https://github.com/jvns/tiny-tls/">on github</a>. I decided to use Go because I heard that Go has good crypto libraries.</p>

<h3 id="the-simplifications">the simplifications</h3>

<p>I only wanted to work on this for a few days at most, so I needed to make some
pretty dramatic simplifications to make it possible to get it done quickly.</p>

<p>I decided my goal was going to be to download this blog&rsquo;s homepage with TLS. So I
don&rsquo;t need to implement a fully general TLS implementation, I just need to
successfully connect to one website.</p>

<p>Specifically, this means that:</p>

<ul>
<li>I only support one cipher suite</li>
<li>I don&rsquo;t verify the server&rsquo;s certificate at all, I just ignore it</li>
<li>my parsing and message formatting can be extremely janky and fragile because I only need to be able to talk to one specific TLS implementation (and believe me, they are)</li>
</ul>

<h3 id="an-amazing-tls-resource-tls13-ulfheim-net">an amazing TLS resource: tls13.ulfheim.net</h3>

<p>Luckily, before starting this I remembered vaguely that I&rsquo;d seen a website that
explained every single byte in a TLS 1.3 connection, with detailed code examples to
reproduce every part. Some googling revealed that it was <a href="https://tls13.ulfheim.net">The New Illustrated TLS Connection</a>.</p>

<p>I can&rsquo;t stress enough how helpful this was, I looked at probably more than a
hundred times and I only looked at the TLS 1.3 RFC for a few small things.</p>

<h3 id="some-cryptography-basics">some cryptography basics</h3>

<p>Before I started working on this, my understanding of TLS was:</p>

<ol>
<li>at the beginning there&rsquo;s some sort of Diffie-Hellman key exchange</li>
<li>you use the key exchange to somehow (how???) get an AES symmetric key and encrypt the rest of the connection with AES</li>
</ol>

<p>This was sort of right, but it turns out it&rsquo;s more complicated than that.</p>

<p>Okay, let&rsquo;s get into my hacky toy TLS implementation. It hopefully goes without saying that you should absolutely not use this code for anything.</p>

<h3 id="step-1-say-hello">step 1: say hello</h3>

<p>First we need to send a &ldquo;Client Hello&rdquo; message. For my purposes this has just 4 pieces of information in it:</p>

<ol>
<li>A randomly generated public key</li>
<li>32 bytes of random data (the &ldquo;Client Random&rdquo;)</li>
<li>The domain name I want to connect to (<code>jvns.ca</code>)</li>
<li>The cipher suites/signature algorithms we want to use (which I just copied
from tls.ulfheim.net). This negotiation process is pretty important in
general but I&rsquo;m ignoring it because I only support one signature algorithm /
cipher suite.</li>
</ol>

<p>The most interesting part of this to me was part 1 &ndash; how do I generate the public key?</p>

<p>I was confused about this for a while but it ended up being just 2 lines of code.</p>

<pre><code>privateKey := random(32)
publicKey, err := curve25519.X25519(privateKey, curve25519.Basepoint)
</code></pre>

<p>You can see the rest of the code to generate the <a href="https://github.com/jvns/tiny-tls/blob/cb5a3665c3487ad1f1d5f917ad069c93dd44967e/format.go#L41">client hello message here</a>
but it&rsquo;s very boring, it&rsquo;s just a lot of bit fiddling.</p>

<h3 id="elliptic-curve-cryptography-is-cool">elliptic curve cryptography is cool</h3>

<p>I am not going to give an explanation of elliptic curve cryptography here,  but I just want to say how point out how cool it is that you can:</p>

<ul>
<li>generate a random 32-byte string as a private key</li>
<li>&ldquo;multiply&rdquo; the private key by the curve&rsquo;s base point to get the public key (this is elliptic curve &ldquo;multiplication&rdquo;, where <code>n * P</code> means &ldquo;add P to itself n times&rdquo;)</li>
<li>that&rsquo;s it!!</li>
</ul>

<p>I wrote &ldquo;multiply&rdquo; in scare quotes because this &ldquo;multiplication&rdquo; doesn&rsquo;t let you
multiply points on the elliptic curve by <em>each other</em>. You can only multiply a
point by an integer.</p>

<p>Here&rsquo;s the function signature  of the <code>X25519</code> function we use to do the
&ldquo;multiplication&rdquo;. You can see one of the arguments is called <code>scalar</code> and one
is called <code>point</code>. And the order of the arguments matters! If you switch them it
won&rsquo;t do the right thing.</p>

<pre><code>func X25519(scalar, point []byte) ([]byte, error)
</code></pre>

<p>I am not going to say more about elliptic curve cryptography here but I love how
simple this is to use &ndash; it seems a lot straightforward than RSA where your
private keys have to be prime numbers.</p>

<p>I don&rsquo;t know if &ldquo;you can use any 32-byte string as a private key&rdquo; is true
for all elliptic curves or just for this specific elliptic curve (<a href="https://en.wikipedia.org/wiki/Curve25519">Curve25519</a>).</p>

<h3 id="step-2-parse-the-server-hello">step 2: parse the server hello</h3>

<p>Next the server says hello. This is very boring, basically we just need to
parse it to get the server&rsquo;s public key which is 32 bytes. <a href="https://github.com/jvns/tiny-tls/blob/cb5a3665c3487ad1f1d5f917ad069c93dd44967e/format.go#L98-L131">Here&rsquo;s the code though</a>.</p>

<h3 id="step-3-calculate-the-keys-to-encrypt-the-handshake">step 3: calculate the keys to encrypt the handshake</h3>

<p>Now that we have the server&rsquo;s public key and we&rsquo;ve sent the server our public
key, we can start to calculate the keys we&rsquo;re going to use to actually encrypt
data.</p>

<p>I was surprised to learn that there are at least 4 different symmetric keys involved in TLS:</p>

<ul>
<li>client handshake key/iv (for the data the client sends in the handshake)</li>
<li>server handshake key/iv (for the data the server sends in the handshaek)</li>
<li>client application key/iv (for the rest of the data the client sends)</li>
<li>server application key/iv (for the rest of the data the server sends)</li>
<li>I think also another key for session resumption, but I didn&rsquo;t implement that</li>
</ul>

<p>We start out by combining the server&rsquo;s public key and our private key to get a
shared secret. This is called &ldquo;elliptic curve diffie hellman&rdquo; or ECDH and it&rsquo;s
pretty simple: &ldquo;multiply&rdquo; the server&rsquo;s private key by our public key:</p>

<pre><code>sharedSecret, err := curve25519.X25519(session.Keys.Private, session.ServerHello.PublicKey)
</code></pre>

<p>This gives us a 32-byte secret key that both the client and the server has. Yay!</p>

<p>But we need 96 bytes (16 + 12) * 4 of keys in total. That&rsquo;s more than 32 bytes!</p>

<h3 id="time-for-key-derivation">time for key derivation</h3>

<p>Apparently the way you turn a small key into more keys is called &ldquo;key
derivation&rdquo;, and TLS 1.3 uses an algorithm called &ldquo;HKDF&rdquo; to do this. I honestly do not
understand this but here is what my code to do it looks like.</p>

<p>It seems to involve alternately calling <code>hkdf.Expand</code> and <code>hkdf.Extract</code> over and
over again a bunch of times.</p>

<pre><code>func (session *Session) MakeHandshakeKeys() {
	zeros := make([]byte, 32)
	psk := make([]byte, 32)
	// ok so far
	if err != nil {
		panic(err)
	}
	earlySecret := hkdf.Extract(sha256.New, psk, zeros) // TODO: psk might be wrong
	derivedSecret := deriveSecret(earlySecret, &quot;derived&quot;, []byte{})
	session.Keys.HandshakeSecret = hkdf.Extract(sha256.New, sharedSecret, derivedSecret)
	handshakeMessages := concatenate(session.Messages.ClientHello.Contents(), session.Messages.ServerHello.Contents())

	cHsSecret := deriveSecret(session.Keys.HandshakeSecret, &quot;c hs traffic&quot;, handshakeMessages)
	session.Keys.ClientHandshakeSecret = cHsSecret
	session.Keys.ClientHandshakeKey = hkdfExpandLabel(cHsSecret, &quot;key&quot;, []byte{}, 16)
	session.Keys.ClientHandshakeIV = hkdfExpandLabel(cHsSecret, &quot;iv&quot;, []byte{}, 12)

	sHsSecret := deriveSecret(session.Keys.HandshakeSecret, &quot;s hs traffic&quot;, handshakeMessages)
	session.Keys.ServerHandshakeKey = hkdfExpandLabel(sHsSecret, &quot;key&quot;, []byte{}, 16)
	session.Keys.ServerHandshakeIV = hkdfExpandLabel(sHsSecret, &quot;iv&quot;, []byte{}, 12)
}
</code></pre>

<p>This was pretty annoying to get working because I kept passing the wrong
arguments to things. The only reason I managed it was because
<a href="https://tls13.ulfheim.net">https://tls13.ulfheim.net</a> provided a bunch of example inputs and outputs and
example code so I was able to write some unit tests and check my code against
the site&rsquo;s example implementation.</p>

<p>Anyway, eventually I got all my keys calculated and it was time to start decrypting!</p>

<h3 id="an-aside-on-ivs">an aside on IVs</h3>

<p>For each key there&rsquo;s also an &ldquo;IV&rdquo; which stands for &ldquo;initialization vector&rdquo;. The
idea seems to be to use a different initialization vector for every message we
encrypt/decrypt, for More Security &trade;.</p>

<p>In this implementation the way we get a different IV for each message is by
xoring the IV with the number of messages sent/received so far.</p>

<h3 id="step-4-write-some-decryption-code">step 4: write some decryption code</h3>

<p>Now that we have all these keys and IVs, we can write a <code>decrypt</code> function.</p>

<p>I thought that TLS just used AES, but apparently it uses something called
&ldquo;authentication encryption&rdquo; on top of AES that I hadn&rsquo;t heard of before.</p>

<p>The wikipedia article explanation of authenticated encryption is actually pretty clear:</p>

<blockquote>
<p>&hellip; authenticated encryption can provide security against <strong>chosen ciphertext</strong> attack. In these attacks, an adversary attempts to gain an advantage against a cryptosystem (e.g., information about the secret decryption key) by submitting carefully chosen ciphertexts to some &ldquo;decryption oracle&rdquo; and analyzing the decrypted results. Authenticated encryption schemes can recognize improperly-constructed ciphertexts and refuse to decrypt them. This, in turn, prevents the attacker from requesting the decryption of any ciphertext unless it was generated correctly using the encryption algorithm</p>
</blockquote>

<p>This makes sense to me because I did some of the cryptopals challenges and there&rsquo;s an attack a bit like this in <a href="https://cryptopals.com/sets/2/challenges/14">cryptopals set 2</a> (I don&rsquo;t know if it&rsquo;s the exact same thing).</p>

<p>Anyway, here&rsquo;s some code that uses authenticated encryption the way the TLS 1.3
spec says it should. I think GCM is an authenticated encryption algorithm.</p>

<pre><code>func decrypt(key, iv, wrapper []byte) []byte {

	block, err := aes.NewCipher(key)
	if err != nil {
		panic(err.Error())
	}

	aesgcm, err := cipher.NewGCM(block)
	if err != nil {
		panic(err.Error())
	}

	additional := wrapper[:5]
	ciphertext := wrapper[5:]

	plaintext, err := aesgcm.Open(nil, iv, ciphertext, additional)
	if err != nil {
		panic(err.Error())
	}
	return plaintext
}
</code></pre>

<h3 id="step-5-decrypt-the-server-handshake">step 5: decrypt the server handshake</h3>

<p>Next the server sends some more handshake data. This contains the certificate
and some other stuff.</p>

<p>Here&rsquo;s my code for decrypting the handshake. Basically it just reads the
encrypted data from the network, decrypts it, and saves it.</p>

<pre><code>record := readRecord(session.Conn)
if record.Type() != 0x17 {
    panic(&quot;expected wrapper&quot;)
}
session.Messages.ServerHandshake = decrypt(session.Keys.ServerHandshakeKey, session.Keys.ServerHandshakeIV, record)
</code></pre>

<p>You might notice that we don&rsquo;t actually <em>parse</em> this data at all &ndash; that&rsquo;s
because we don&rsquo;t need the contents, since we&rsquo;re not verifying the server&rsquo;s
certificate.</p>

<p>I was surprised that you don&rsquo;t technically need to look at the server&rsquo;s
certificate at all to make a TLS connection (though obviously you should verify it!). I thought you would need to at least parse it to get a
key out of it or something.</p>

<p>We do need to be able to hash the handshake for the next step though, so we
have to store it.</p>

<h3 id="step-6-derive-more-keys">step 6: derive more keys</h3>

<p>We use a hash of the SHA256 handshake data we just got from the server to
generate even more symmetric keys. This is almost the last step!</p>

<p>This is almost exactly the same as the key derivation code from before, but I&rsquo;m
including it because I was surprised by how much work needed to be done to generate all these keys.</p>

<pre><code>func (session *Session) MakeApplicationKeys() {
    handshakeMessages := concatenate(
        session.Messages.ClientHello.Contents(),
        session.Messages.ServerHello.Contents(),
        session.Messages.ServerHandshake.Contents())

    zeros := make([]byte, 32)
    derivedSecret := deriveSecret(session.Keys.HandshakeSecret, &quot;derived&quot;, []byte{})
    masterSecret := hkdf.Extract(sha256.New, zeros, derivedSecret)

    cApSecret := deriveSecret(masterSecret, &quot;c ap traffic&quot;, handshakeMessages)
    session.Keys.ClientApplicationKey = hkdfExpandLabel(cApSecret, &quot;key&quot;, []byte{}, 16)
    session.Keys.ClientApplicationIV = hkdfExpandLabel(cApSecret, &quot;iv&quot;, []byte{}, 12)

    sApSecret := deriveSecret(masterSecret, &quot;s ap traffic&quot;, handshakeMessages)
    session.Keys.ServerApplicationKey = hkdfExpandLabel(sApSecret, &quot;key&quot;, []byte{}, 16)
    session.Keys.ServerApplicationIV = hkdfExpandLabel(sApSecret, &quot;iv&quot;, []byte{}, 12)
}
</code></pre>

<h3 id="step-7-finish-the-handshake">step 7: finish the handshake</h3>

<p>Next we need to send a &ldquo;handshake finished&rdquo; message to the server to verify that everything is done. That code is <a href="https://github.com/jvns/tiny-tls/blob/cb5a3665c3487ad1f1d5f917ad069c93dd44967e/crypto.go#L177-L183">here</a>.</p>

<p>And now we&rsquo;re done the handshake! That was the hard part, sending and receiving
the data is relatively easy.</p>

<h3 id="step-8-make-a-http-request">step 8: make a HTTP request</h3>

<p>I wrote a <code>SendData</code> function that encrypts and sends data using our keys. This time we&rsquo;re using the &ldquo;application&rdquo; keys and not the handshake keys. This made making a HTTP request pretty simple:</p>

<pre><code>req := fmt.Sprintf(&quot;GET / HTTP/1.1\r\nHost: %s\r\n\r\n&quot;, domain)
session.SendData([]byte(req))
</code></pre>

<h3 id="step-9-we-can-actually-decrypt-the-response">step 9: we can actually decrypt the response!!!</h3>

<p>Now comes the moment I&rsquo;d been waiting for &mdash; actually decrypting the response
from the server!!! But here I needed to learn something else about TLS.</p>

<h3 id="tls-data-comes-in-blocks">TLS data comes in blocks</h3>

<p>I previously thought that once you established the connection, encrypted TLS
data was just a stream. But that&rsquo;s not how it works &ndash; instead, it&rsquo;s
transmitted in blocks. Like, you&rsquo;ll get a chunk of ~1400 bytes to decrypt, and
then another chunk, and then another chunk.</p>

<p>I&rsquo;m not sure why the blocks have the size they do (maybe it&rsquo;s so that each one will fit inside a TCP
packet ???), but in theory I think they could be up to 65535 bytes, since their
size field is 2 bytes. The blocks I got were all 1386 bytes each.</p>

<p>Every time we get a block, we have to:</p>

<ul>
<li>calculate a new IV as <code>old_iv xor num_records_received</code></li>
<li>decrypt it using the key and the new IV</li>
<li>increment the count of records received</li>
</ul>

<p>Here&rsquo;s what the <code>ReceiveData()</code> function I wrote looks like.</p>

<p>The most interesting part of this is the <code>iv[11] ^= session.RecordsReceived</code> &ndash;
that&rsquo;s the part that adjusts the IV for each block.</p>

<pre><code>func (session *Session) ReceiveData() []byte {
	record := readRecord(session.Conn)
	iv := make([]byte, 12)
	copy(iv, session.Keys.ServerApplicationIV)
	iv[11] ^= session.RecordsReceived
	plaintext := decrypt(session.Keys.ServerApplicationKey, iv, record)
	session.RecordsReceived += 1
	return plaintext
}
</code></pre>

<p>This <code>iv[11]</code> thing assumes that there are less than 255 blocks which obviously
is not true in general in TLS, but I was lazy and to download my blog&rsquo;s
homepage I only needed 82 blocks.</p>

<p>We actually have to do this when we send data too, but I didn&rsquo;t implement it
because we only sent 1 packet.</p>

<h3 id="problem-getting-the-whole-block-of-tls-data">problem: getting the whole block of tLS data</h3>

<p>I ran into one problem with TCP where sometimes I&rsquo;d try to read a block of TLS
data (~1386 bytes), but I wouldn&rsquo;t get the whole thing. I guess the TLS blocks
can be split up across multiple TCP packets.</p>

<p>I fixed this in a really dumb way, by just polling the TCP connection in a loop
until it gave me the data I wanted. Here&rsquo;s my code to do that:</p>

<pre><code>func read(length int, reader io.Reader) []byte {
	var buf []byte
	for len(buf) != length {
		buf = append(buf, readUpto(length-len(buf), reader)...)
	}
	return buf
}
</code></pre>

<p>I assume a real TLS implementation would use a thread pool or coroutines or
something to manage this.</p>

<h3 id="step-10-knowing-when-we-re-done">step 10: knowing when we&rsquo;re done</h3>

<p>When the HTTP response is done, we get these bytes: <code>[]byte{48, 13, 10, 13, 10, 23}</code>.
This seems to be because my HTTP server is using chunked transfer encoding, so
there&rsquo;s no <code>Content-Length</code> header and I need to watch for those bytes at the
end instead.</p>

<p>So here&rsquo;s the code to receive the HTTP response. Basically we just loop until
we see those bytes, then we stop.</p>

<pre><code>func (session *Session) ReceiveHTTPResponse() []byte {
	var response []byte
	for {
		pt := session.ReceiveData()
		if string(pt) == string([]byte{48, 13, 10, 13, 10, 23}) {
			break
		}
		response = append(response, pt...)
	}
	return response
}
</code></pre>

<h3 id="that-s-it">that&rsquo;s it!</h3>

<p>Finally, I ran the program and I downloaded my blog&rsquo;s homepage! It worked! Here&rsquo;s what the results look like:</p>

<pre><code>$ go build; ./tiny-tls
HTTP/1.1 200 OK
Date: Wed, 23 Mar 2022 19:37:47 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
... lots more headers and HTML follow...
</code></pre>

<p>Okay, the results are kind of anticlimactic, it&rsquo;s just the same as what you&rsquo;d
see if you ran <code>curl -i https://jvns.ca</code> except with no formatting. But I was
extremely excited when I saw it.</p>

<h3 id="unit-tests-are-great">unit tests are great</h3>

<p>Every time I write networking code like this, I forget that unit testing is
good, and I thrash around with a bunch of parsing / formatting code that does
not work and just getting NOPE messages back from the server on the other end.</p>

<p>And then I remember unit tests. In this case, I copied a bunch of the data from
the <a href="https://tls13.ulfheim.net">https://tls13.ulfheim.net</a> example and put it into my unit tests so that I
could quickly make sure that my parsing and crypto were working correctly.
It made everything about 10 times easier and faster.</p>

<h3 id="some-things-i-learned">some things I learned</h3>

<p>This was really fun! I learned that</p>

<ul>
<li>elliptic curve diffie-hellman is very cool, and at least with Curve25519 you can use literally any 32-byte string as a private key</li>
<li>there are a LOT of different symmetric keys involved in TLS and the key derivation process is pretty complicated</li>
<li>TLS uses AES with some extra &ldquo;authenticated encryption&rdquo; algorithms on top</li>
<li>TLS data is sent/received as a bunch of blocks, not as a stream</li>
</ul>

<p>My code truly is terrible, it can connect to my site (<code>jvns.ca</code>) and I think literally no other sites.</p>

<p>I won&rsquo;t pretend to understand all the reasons TLS is designed this way, but it
was a fun way to spend a couple of days, I feel a little more informed, and I
think it&rsquo;ll be easier for me to understand things I read about TLS in the
future.</p>

<h3 id="a-plug-for-cryptopals">a plug for cryptopals</h3>

<p>If you want to learn about cryptography and you haven&rsquo;t tried the
<a href="https://cryptopals.com/">cryptopals</a> challenges, I really recommend them &ndash; you
get to implement a lot of attacks on crypto systems and it&rsquo;s very fun.</p>


guid: https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/
title: A list of new(ish) command line tools
url: https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! Today I asked <a href="https://twitter.com/b0rk/status/1513903221466664962">on twitter</a> about newer
command line tools, like <code>ripgrep</code> and <code>fd</code> and <code>fzf</code> and <code>exa</code> and <code>bat</code>.</p>

<p>I got a bunch of replies with tools I hadn&rsquo;t heard of, so I thought I&rsquo;d make a
list here. A lot of people also pointed at the <a href="https://github.com/ibraheemdev/modern-unix">modern-unix</a> list.</p>

<h3 id="replacements-for-standard-tools">replacements for standard tools</h3>

<ul>
<li><a href="https://github.com/BurntSushi/ripgrep/">ripgrep</a>, <a href="https://github.com/ggreer/the_silver_searcher">ag</a>, <a href="https://github.com/beyondgrep/ack3">ack</a> (grep)</li>
<li><a href="https://github.com/ogham/exa">exa</a>, <a href="https://github.com/Peltoche/lsd">lsd</a> (ls)</li>
<li><a href="https://mosh.org/">mosh</a> (ssh)</li>
<li><a href="https://github.com/sharkdp/bat">bat</a> (cat)</li>
<li><a href="https://github.com/dandavison/delta">delta</a> (a pager for git)</li>
<li><a href="https://github.com/sharkdp/fd">fd</a> (find)</li>
<li><a href="https://www.nlnetlabs.nl/projects/ldns/about/">drill</a>, <a href="https://github.com/ogham/dog">dog</a> (dig)</li>
<li><a href="https://github.com/muesli/duf">duf</a> (df)</li>
<li><a href="https://github.com/bootandy/dust">dust</a>, ncdu (du)</li>
<li><a href="https://www.pgcli.com/">pgcli</a> (psql)</li>
<li><a href="https://github.com/ClementTsang/bottom">btm</a>, <a href="https://github.com/aristocratos/btop">btop</a>, <a href="https://github.com/nicolargo/glances">glances</a>, <a href="https://github.com/aksakalli/gtop">gtop</a>, <a href="https://github.com/bvaisvil/zenith">zenith</a> (top)</li>
<li><a href="https://tldr.sh/">tldr</a> (man, sort of)</li>
<li><a href="https://github.com/chmln/sd">sd</a> (sed)</li>
<li><a href="https://github.com/Wilfred/difftastic">difftastic</a> (diff)</li>
<li>mtr (traceroute)</li>
<li><a href="https://plocate.sesse.net/">plocate</a> (locate)</li>
<li>xxd, <a href="https://github.com/sharkdp/hexyl">hexyl</a> (hexdump)</li>
</ul>

<h3 id="new-inventions">new inventions</h3>

<p>Here are some tools that are not exactly replacements for standard tools:</p>

<ul>
<li><a href="https://github.com/rupa/z">z</a>, <a href="https://github.com/clvv/fasd">fasd</a>, <a href="https://github.com/wting/autojump">autojump</a>, <a href="https://github.com/ajeetdsouza/zoxide">zoxide</a> (tools to make it easier to find files / change directories)</li>
<li><a href="https://github.com/Canop/broot">broot</a>, <a href="https://github.com/jarun/nnn">nnn</a>, <a href="https://github.com/ranger/ranger">ranger</a> (file manager)</li>
<li><a href="https://github.com/direnv/direnv">direnv</a> (load environment variables depending on the current directory)</li>
<li><a href="https://github.com/junegunn/fzf">fzf</a>, <a href="https://github.com/peco/peco">peco</a> (&ldquo;fuzzy finder&rdquo;)</li>
<li><a href="https://github.com/schollz/croc">croc</a> and <a href="https://github.com/magic-wormhole/magic-wormhole">magic-wormhole</a> (send files from one computer to another)</li>
<li><a href="https://github.com/sharkdp/hyperfine">hyperfine</a> (benchmarking)</li>
<li><a href="https://httpie.io/">httpie</a>, <a href="https://github.com/rs/curlie">curlie</a>, <a href="https://github.com/ducaale/xh">xh</a> (for making HTTP requests)</li>
<li><a href="https://github.com/eradman/entr">entr</a> (run arbitrary commands when files change)</li>
<li><a href="https://github.com/asdf-vm/asdf">asdf</a> (version manager for multiple languages)</li>
<li><a href="https://github.com/jonas/tig">tig</a>, <a href="https://github.com/jesseduffield/lazygit">lazygit</a> (interactive interfaces for git)</li>
<li><a href="https://github.com/jesseduffield/lazydocker">lazydocker</a> (interactive interface for docker)</li>
<li><a href="https://github.com/theryangeary/choose">choose</a> (the basics of awk/cut)</li>
<li><a href="https://github.com/bcicen/ctop">ctop</a> (top for containers)</li>
<li><a href="https://github.com/nvbn/thefuck">fuck</a> (autocorrect command line errors)</li>
<li><a href="https://tmate.io/">tmate</a> (share your terminal with a friend)</li>
<li><a href="https://github.com/tstack/lnav">lnav</a>, <a href="https://github.com/rcoh/angle-grinder">angle-grinder</a> (tools for managing logs)</li>
<li><a href="https://github.com/visit1985/mdp">mdp</a>, <a href="https://github.com/charmbracelet/glow">glow</a> (ways to display markdown in the terminal)</li>
<li>pbcopy/pbpaste (for clipboard &lt;&gt; stdin/stdout) maybe aren&rsquo;t &ldquo;new&rdquo; but were mentioned a lot. You can <a href="https://stackoverflow.com/questions/5130968/how-can-i-copy-the-output-of-a-command-directly-into-my-clipboard/41843618#41843618">use xclip</a> to do the same thing on Linux.</li>
</ul>

<p><strong>JSON/YAML/CSV things:</strong></p>

<ul>
<li><a href="https://stedolan.github.io/jq/">jq</a> (a great JSON-wrangling tool)</li>
<li><a href="https://github.com/kellyjonbrazil/jc">jc</a> (convert various tools&rsquo; output into JSON)</li>
<li><a href="https://github.com/jpmens/jo">jo</a> (create JSON objects)</li>
<li><a href="https://github.com/mikefarah/yq">yq</a>  (like <code>jq</code>, but for YAML). there&rsquo;s also <a href="https://github.com/kislyuk/yq">another yq</a></li>
<li><a href="https://github.com/wader/fq">fq</a> (like <code>jq</code>, but for binary)</li>
<li><a href="https://github.com/mgdm/htmlq">htmlq</a> (like <code>jq</code>, but for HTML)</li>
<li><a href="https://github.com/antonmedv/fx">fx</a> (interactive json tool)</li>
<li><a href="https://github.com/PaulJuliusMartinez/jless">jless</a> (json pager)</li>
<li><a href="https://github.com/BurntSushi/xsv">xsv</a> (a command line tool for csv files, from burntsushi)</li>
<li><a href="https://www.visidata.org/">visidata</a> (&ldquo;an interactive multitool for tabular data&rdquo;)</li>
<li><a href="https://github.com/johnkerl/miller">miller</a> (&ldquo;like awk/sed/cut/join/sort for CSV/TSV/JSON/JSON lines&rdquo;)</li>
</ul>

<p><strong>grep things:</strong></p>

<ul>
<li><a href="https://pdfgrep.org/">pdfgrep</a> (grep for PDF)</li>
<li><a href="https://github.com/tomnomnom/gron">gron</a> (make JSON greppable)</li>
<li><a href="https://github.com/phiresky/ripgrep-all">ripgrep-all</a> (ripgrep, but also PDF, zip, ebooks, etc)</li>
</ul>

<h3 id="less-new-tools">less-new tools</h3>

<p>Here are a few of not-so-new tools folks mentioned aren&rsquo;t that well known:</p>

<ul>
<li>pv (&ldquo;pipe viewer&rdquo;, gives you a progress bar for a pipe)</li>
<li>vidir (from <a href="https://joeyh.name/code/moreutils">moreutils</a>, lets you batch rename/delete files in vim)</li>
<li>sponge, ts, parallel (also from moreutils)</li>
</ul>

<h3 id="some-of-my-favourites">some of my favourites</h3>

<p>My favourites of these that I use already are <code>entr</code>, <code>ripgrep</code>, <code>git-delta</code>,
<code>httpie</code>, <code>plocate</code>, and <code>jq</code>.</p>

<p>I&rsquo;m interested in trying out <code>direnv</code>, <code>btm</code>, <code>z</code>, <code>xsv</code>, and <code>duf</code>, but I
think the most exciting tool I learned about is <code>vidir</code>.</p>


guid: https://jvns.ca/blog/2022/04/26/new-zine--how-dns-works-/
title: New zine: How DNS Works!
url: https://jvns.ca/blog/2022/04/26/new-zine--how-dns-works-/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! On Thursday we released a new zine about one of my favourite computer systems: DNS!</p>

<p>You can get it for $12 here:
<a href="https://wizardzines.com/zines/dns">https://wizardzines.com/zines/dns</a>, or get
an <a href="https://wizardzines.com/zines/all-the-zines/">11-pack of all my zines here</a>.</p>

<p>Here&rsquo;s the cover and table of contents:</p>

<div align="center">
<a href="https://wizardzines.com/zines/dns"><img width="400px" src="https://jvns.ca/images/dns-cover.png"></a>
<a href="https://jvns.ca/images/dns-toc.png"><img width="400px" src="https://jvns.ca/images/dns-toc.png"></a>
</div>

<h3 id="why-dns">why DNS?</h3>

<p>I wanted to write about DNS for three reasons:</p>

<ol>
<li><strong>DNS is everywhere!</strong> You basically can’t use the internet without using
DNS — nobody is going to memorize all the IP addresses for every domain they
use. You need to look them up with DNS!</li>
<li><strong>DNS has a really cool decentralized design!</strong> I love that it gives you
control of your own little corner of the internet (it just costs ~$12/year to
own a domain!), I love that the basic design is relatively straightforward,
and I love that it’s worked basically the same way for 40 years and scaled
relatively well over time</li>
<li><strong>DNS is very frustrating!</strong> I’ve run into some VERY weird DNS problems over
the years, and it can feel magical and incomprehensible if you don’t know
how it works. But once you learn how DNS works, these problems all become
totally possible to understand.</li>
</ol>

<p>And I&rsquo;ve never seen a clear explanation of DNS that included all of the
technical details I wanted to know, so I decided to write one.</p>

<h3 id="a-dns-playground">a DNS playground!</h3>

<p>I find the most fun way to learn is by experimenting and breaking things! So back in December,
<a href="https://marieflanagan.com/">Marie</a> and I built a playground where you can do
weird DNS experiments with no consequences if you mess something up.</p>

<p>It’s called <a href="https://messwithdns.net">Mess With DNS</a>. It has examples of
experiments you can try, and you’ve very encouraged to come up with your own
experiments.</p>

<p>There&rsquo;s also <a href="https://jvns.ca/blog/2021/12/15/mess-with-dns/">a blog post</a> from when we released it in December.</p>

<h3 id="a-simple-dns-lookup-tool">a simple DNS lookup tool</h3>

<p>I also built a little DNS lookup tool at <a href="https://dns-lookup.jvns.ca">https://dns-lookup.jvns.ca</a> to provide
an easy way to do DNS queries without using the command line. Even though I
love the command line I actually find myself using it all the time.</p>

<h3 id="what-we-left-out-dns-security">what we left out: DNS security</h3>

<p>The main thing that <em>isn&rsquo;t</em> in the zine is DNS security (DNS over HTTPS, DNS
over TLS, DNSSEC).  This is because the DNS security landscape still seems to
be evolving, and I didn&rsquo;t feel confident that we could write something that
would still be true in 5-10 years.</p>

<p>I think I&rsquo;ll try to write a blog post about DNS security at some point though.</p>

<h3 id="identifying-common-points-of-confusion-is-amazing">identifying common points of confusion is amazing</h3>

<p>I mentioned earlier that my friend <a href="https://marieflanagan.com/">Marie Claire LeBlanc Flanagan</a> and I built
Mess With DNS together in December. That was really fun, so Marie and I decided
to work together on this zine too &ndash; we paired on it for about an hour every
weekday for almost 4 months. I&rsquo;m pretty sure I wouldn&rsquo;t have finished the zine
without her.</p>

<p>She was incredibly helpful with everything to do with the zine (editing!
character design! marketing! rewriting the NS records page 5 times!), but one
of the biggest things she improved for this zine was <em>feedback</em>.</p>

<p>Usually I only get feedback from maybe 10 beta readers, but this time we wanted
to hear from more people and get better at identifying common points of confusion.</p>

<p>We did this by building a small custom feedback website where people could
click on a page and leave feedback (&ldquo;I have a question! This is confusing! I
learned something! I have a suggestion! I love this!&ldquo;).</p>

<p>Building a custom website let us organize the feedback way more easily and hear
from more people &ndash; in the end we got ~1200 feedback items from ~60 people and
all of the feedback made the zine a LOT better.</p>

<p>I might write a longer meta post about all of this later.</p>

<h3 id="some-blog-posts-leading-up-to-this-zine">some blog posts leading up to this zine</h3>

<p>I wrote a bunch of blog posts about DNS over the last year or so while thinking
about this zine. Here they all are:</p>

<ul>
<li><a href="https://jvns.ca/blog/how-updating-dns-works/">What happens when you update your DNS?</a></li>
<li><a href="https://jvns.ca/blog/2021/02/24/a-little-tool-to-make-dns-queries/">A little tool to make DNS queries</a></li>
<li><a href="https://jvns.ca/blog/2021/03/31/dnspeep-tool/">A tool to spy on your DNS queries: dnspeep</a></li>
<li><a href="https://jvns.ca/blog/2021/11/04/how-do-you-tell-if-a-problem-is-caused-by-dns/">How do you tell if a problem is caused by DNS?</a></li>
<li><a href="https://jvns.ca/blog/2021/12/06/dns-doesn-t-propagate/">DNS &ldquo;propagation&rdquo; is actually caches expiring</a></li>
<li><a href="https://jvns.ca/blog/2022/01/05/why-might-you-run-your-own-dns-server-/">Why might you run your own DNS server?</a></li>
<li><a href="https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/">Some ways DNS can break</a></li>
<li><a href="https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/">A toy DNS resolver</a></li>
<li><a href="https://jvns.ca/blog/2022/02/14/some-dns-terminology/">The multiple meanings of &ldquo;nameserver&rdquo; and &ldquo;DNS resolver&rdquo;</a></li>
</ul>

<h3 id="you-can-get-a-print-copy-shipped-to-you">you can get a print copy shipped to you!</h3>

<p>There&rsquo;s always been the option to print the zines yourself on your home printer.</p>

<p>But this time there&rsquo;s a new option too: you can get a print copy shipped to
you! (just click on the &ldquo;print version&rdquo; link on <a href="https://wizardzines.com/zines/dns/">this
page</a>)</p>

<p>The only caveat is that the international shipping costs are unreasonably high,
so if you&rsquo;re outside the US it probably doesn&rsquo;t make sense to get just 1 zine
shipped to you. I&rsquo;m still trying to find a workaround for that. Orders of $60+
do include free international shipping though.</p>

<p>Here&rsquo;s the link to get the zine again:
<a href="https://wizardzines.com/zines/dns/">https://wizardzines.com/zines/dns/</a></p>

<h3 id="thank-you">thank you</h3>

<p>If you&rsquo;ve bought zines in the past, thanks so much for all your support over
the years. None of this would be possible without you.</p>


guid: https://jvns.ca/blog/2022/05/10/pages-that-didn-t-make-it-into--how-dns-works-/
title: Pages that didn't make it into "How DNS Works"
url: https://jvns.ca/blog/2022/05/10/pages-that-didn-t-make-it-into--how-dns-works-/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! A couple weeks ago I released a new zine called <a href="https://wizardzines.com/zines/dns/">How DNS Works</a>.</p>

<p>When I started writing that zine (in, uh, January 2021), I originally had in
mind a broader zine on &ldquo;everything you need to know to own a domain&rdquo;. So it had
a bunch of pages on domain registration, TLS, and email.</p>

<p>At the time I thought &ldquo;I can just explain DNS in like 5 pages, it&rsquo;s not
that complicated, there will be lots of space for other topics about domains&rdquo;.
I was extremely wrong about that and it turned out I needed all 28 pages to
explain DNS. So I ended up deciding to just focus the zine on DNS and all those
other topics didn&rsquo;t make it into the final zine.</p>

<p>This morning it occurred to me that instead of letting all of the old draft
pages languish in purgatory on my hard drive, I could post those extra pages
here all together on my blog. So here they are!</p>

<h3 id="disclaimer-not-super-cohesive">disclaimer: not super cohesive</h3>

<p>I will say (as a disclaimer) that these pages aren&rsquo;t as cohesive as I usually
like my zines to be and they definitely do not tell you everything you need to
need to know to own a domain.</p>

<h3 id="domain-registration">domain registration</h3>

<p><img src="https://jvns.ca/images/domain-extras/buying-domain.png">
<img src="https://jvns.ca/images/domain-extras/registrar.png">
<img src="https://jvns.ca/images/domain-extras/domain-expiry.png">
<img src="https://jvns.ca/images/domain-extras/transfers.png">
<img src="https://jvns.ca/images/domain-extras/domain-privacy.png">
<img src="https://jvns.ca/images/domain-extras/tld.png"></p>

<h3 id="email">email</h3>

<p><img src="https://jvns.ca/images/domain-extras/receiving-email.png">
<img src="https://jvns.ca/images/domain-extras/sending-email.png"></p>

<p>I should say that these 2 pages don&rsquo;t really do email justice &ndash; email security
is a HUGE topic that honestly I don&rsquo;t know a lot about.</p>

<h3 id="tls">TLS</h3>

<p><img src="https://jvns.ca/images/domain-extras/why-tls-matters.png">
<img src="https://jvns.ca/images/domain-extras/tls-certificates.png"></p>

<p>These two pages also don&rsquo;t remotely cover TLS, it&rsquo;s possible I&rsquo;ll write more in
depth about TLS at some point. Who knows!</p>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>though I will say: if you liked these, you might be interested in buying <a href="https://wizardzines.com/zines/dns/">How DNS Works</a> :)</p>


guid: https://jvns.ca/blog/2022/05/12/sqlite-utils--a-nice-way-to-import-data-into-sqlite/
title: sqlite-utils: a nice way to import data into SQLite for analysis
url: https://jvns.ca/blog/2022/05/12/sqlite-utils--a-nice-way-to-import-data-into-sqlite/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! This is a quick post about a nice tool I found recently called <a href="https://sqlite-utils.datasette.io">sqlite-utils</a>, from the <a href="https://jvns.ca/#cool-computer-tools---features---ideas">tools category</a>.</p>

<p>Recently I wanted to do some basic data analysis using data from my Shopify
store. So I figured I&rsquo;d query the Shopify API and import my data into SQLite,
and then I could make queries to get the graphs I want.</p>

<p>But this seemed like a lot of boring work, like I&rsquo;d have to write a
schema and write a Python program. So I hunted around for a solution, and I
found <code>sqlite-utils</code>, a tool designed to make it easy to import arbitrary data
into SQLite to do data analysis on the data.</p>

<h3 id="sqlite-utils-automatically-generates-a-schema">sqlite-utils automatically generates a schema</h3>

<p>The Shopify data has about a billion fields and I really did not want to type
out a schema for it. <code>sqlite-utils</code> solves this problem: if I have an array of
JSON orders, I can create a new SQLite table with that data in it like this:</p>

<pre><code>import sqlite_utils

orders = ... # (some code to get the `orders` array here)

db = sqlite_utils.Database('orders.db')
db['shopify_orders'].insert_all(orders)
</code></pre>

<h3 id="you-can-alter-the-schema-if-there-are-new-fields-with-alter">you can alter the schema if there are new fields (with <code>alter</code>)</h3>

<p>Next, I ran into a problem where on the 5th page of downloads, the JSON
contained a new field that I hadn&rsquo;t seen before.</p>

<p>Luckily, <code>sqlite-utils</code> thought of that: there&rsquo;s an <code>alter</code> flag which will
update the table&rsquo;s schema to include the new fields.  ```</p>

<p>Here&rsquo;s what the code for that looks like</p>

<pre><code>db['shopify_orders'].insert_all(orders, alter=True)
</code></pre>

<h3 id="you-can-deduplicate-existing-rows-with-upsert">you can deduplicate existing rows (with <code>upsert</code>)</h3>

<p>Next I ran into a problem where sometimes when doing a sync, I&rsquo;d download data
from the API where some of it was new and some wasn&rsquo;t.</p>

<p>So I wanted to do an &ldquo;upsert&rdquo; where it only created new rows if the item didn&rsquo;t
already exist. <code>sqlite-utils</code> also thought of this, and there&rsquo;s an <code>upsert</code>
method.</p>

<p>For this to work you have to specify the primary key. For me that was
<code>pk=&quot;id&quot;</code>. Here&rsquo;s what my final code looks like:</p>

<pre><code>db['shopify_orders'].upsert_all(
    orders,
    pk=&quot;id&quot;,
    alter=True
)
</code></pre>

<h3 id="there-s-also-a-command-line-tool">there&rsquo;s also a command line tool</h3>

<p>I&rsquo;ve talked about using <code>sqlite-utils</code> as a library so far, but there&rsquo;s also a
command line tool which is really useful.</p>

<p>For example, this inserts the data from a <code>plants.csv</code> into a <code>plants</code> table:</p>

<pre><code>sqlite-utils insert plants.db plants plants.csv --csv
</code></pre>

<h3 id="format-conversions">format conversions</h3>

<p>I haven&rsquo;t tried this yet, but here&rsquo;s a cool example from the help docs of how
you can do format conversions, like converting a string to a float:</p>

<pre><code>sqlite-utils insert plants.db plants plants.csv --csv --convert '
return {
  &quot;name&quot;: row[&quot;name&quot;].upper(),
  &quot;latitude&quot;: float(row[&quot;latitude&quot;]),
  &quot;longitude&quot;: float(row[&quot;longitude&quot;]),
}'
</code></pre>

<p>This seems really useful for CSVs, where by default it&rsquo;ll often interpret numeric
data as strings if you don&rsquo;t do this conversions.</p>

<h3 id="metabase-seems-nice-too">metabase seems nice too</h3>

<p>Once I had all the data in SQLite, I needed a way to draw graphs with it. I
wanted some dashboards, so I ended up using <a href="https://www.metabase.com/">Metabase</a>, an open source business
intelligence tool. I found it very straightforward and it seems like a really
easy way to turn SQL queries into graphs.</p>

<p>This whole setup (sqlite-utils + metabase + SQL) feels a lot easier to use than
my previous setup, where I had a custom Flask website that used plotly and
pandas to draw graphs.</p>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>I was really delighted by <code>sqlite-utils</code>, it was super easy to use and it did
everything I wanted.</p>


guid: https://jvns.ca/blog/2022/06/28/some-notes-on-bubblewrap/
title: Notes on running containers with bubblewrap
url: https://jvns.ca/blog/2022/06/28/some-notes-on-bubblewrap/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! About a year ago I got mad about Docker container startup time. This was
because I was building an <a href="https://jvns.ca/blog/2021/09/24/new-tool--an-nginx-playground/">nginx playground</a>
where I was starting a new &ldquo;container&rdquo; on every HTTP request, and so for it to
feel reasonably snappy, nginx needed to start quickly.</p>

<p>Also, I was running this project on a pretty small cloud machine (256MB RAM), a
small CPU, so I really wanted to avoid unnecessary overhead.</p>

<p>I&rsquo;ve been looking for a way to run containers faster since then, but I couldn&rsquo;t
find one until last week when I discovered
<a href="https://github.com/containers/bubblewrap">bubblewrap</a>!! It&rsquo;s very fast and I
think it&rsquo;s super cool, but I also ran into a bunch of fun problems that I
wanted to write down for my future self.</p>

<h3 id="some-disclaimers">some disclaimers</h3>

<ul>
<li>I&rsquo;m not sure if the way I&rsquo;m using bubblewrap in this post is maybe not how it&rsquo;s intended to be used</li>
<li>there are a lot of sharp edges when using bubblewrap in this way, you need to
think a lot about Linux namespaces and how containers work</li>
<li>bubblewrap is a security tool but I am not a security person and I am only
doing this for weird tiny projects. you should definitely not take security
advice from me.</li>
</ul>

<p>Okay, all of that said, let&rsquo;s talk about I&rsquo;m trying to use bubblewrap to run
containers fast and in a relatively secure way :)</p>

<h3 id="docker-containers-take-300ms-to-start-on-my-machine">Docker containers take ~300ms to start on my machine</h3>

<p>I ran a quick benchmark to see how long a Docker container takes to run a
simple command (<code>ls</code>). For both Docker and Podman, it&rsquo;s about 300ms.</p>

<pre><code>$ time docker run --network none -it ubuntu:20.04 ls / &gt; /dev/null
Executed in  378.42 millis 
$ time podman run --network none -it ubuntu:20.04 ls / &gt; /dev/null
Executed in  279.27 millis
</code></pre>

<p>Almost all of this time is overhead from docker and podman &ndash; just running <code>ls</code>
by itself takes about 3ms:</p>

<pre><code>$ time ls / &gt; /dev/null
Executed in    2.96 millis 
</code></pre>

<p>I want to stress that, while I&rsquo;m not sure exactly what the slowest part of
Docker and podman startup time is (I spent 5 minutes trying to profile them and
gave up), I&rsquo;m 100% sure it&rsquo;s something important.</p>

<p>The way we&rsquo;re going to run containers faster with bubblewrap has a lot of
limitations and it&rsquo;s a lower level interface which is a lot trickier to use.</p>

<h3 id="goal-1-containers-that-start-quickly">goal 1: containers that start quickly</h3>

<p>I felt like it <em>should</em> be possible to have containers that start essentially
instantly or at least in less than 5ms. My thought process:</p>

<ul>
<li>creating a new namespace with <code>unshare</code> is basically instant</li>
<li><a href="https://jvns.ca/blog/2016/10/10/what-even-is-a-container/">containers are basically just a bunch of namespaces</a></li>
<li>what&rsquo;s the problem?</li>
</ul>

<h3 id="container-startup-time-is-usually-not-that-important">container startup time is (usually) not that important</h3>

<p>Most of the time when people are using containers, they&rsquo;re running some
long-running process inside the container like a webserver, so it doesn&rsquo;t
really matter if it takes 300ms to start.</p>

<p>So it makes sense to me that there aren&rsquo;t a lot of container tools that
optimize for startup time. But I still wanted to optimize for startup time :)</p>

<h3 id="goal-2-run-the-containers-as-an-unprivileged-user">goal 2: run the containers as an unprivileged user</h3>

<p>Another goal I had was to be able to run my containers as an unprivileged user
instead of root.</p>

<p>I was surprised the first time I learned that Docker actually runs containers
as root &ndash; even though I run <code>docker run ubuntu:20.04</code> as an unprivileged user (<code>bork</code>), that
message is actually sent to a daemon running as root, and the Docker container
process itself also runs as root (albeit a <code>root</code> that&rsquo;s stripped of all its
capabilities).</p>

<p>That&rsquo;s fine for Docker (they have lots of very smart people making sure that
they get it right!), but if I&rsquo;m going to do container stuff <em>without</em> using
Docker (for the speed reasons mentioned above), I&rsquo;d rather not do it as root to
keep everything a bit more secure.</p>

<h3 id="podman-can-run-containers-as-an-non-root-user">podman can run containers as an non-root user</h3>

<p>Before we start talking about how to do weird stuff with bubblewrap, I want to
quickly talk about a much more normal tool to run containers: podman!</p>

<p>Podman, unlike Docker, can run containers as an unprivileged user!</p>

<p>If I run this from my normal user:</p>

<pre><code>$ podman run -it ubuntu:20.04 ls
</code></pre>

<p>it doesn&rsquo;t secretly run as root behind the scenes! It just starts the container
as my normal user, and then uses something called &ldquo;user namespaces&rdquo; so that
<em>inside the container</em> I appear to be root.</p>

<p>The other cool thing aboud podman is that it has exactly the same interface as
Docker, so you can just take a Docker command and replace <code>docker</code> with
<code>podman</code> and it&rsquo;ll Just Work. I&rsquo;ve found that sometimes I need to do some extra
work to get podman to work in practice, but it&rsquo;s still pretty nice that it has
the same command line interface.</p>

<p>This &ldquo;run containers as a non-root user&rdquo; feature is normally called &ldquo;rootless
containers&rdquo;. (I find that name kind of counterintuitive, but that&rsquo;s what people call it)</p>

<h3 id="failed-attempt-1-write-my-own-tool-using-runc">failed attempt 1: write my own tool using <code>runc</code></h3>

<p>I knew that Docker and podman use
<a href="https://github.com/opencontainers/runc">runc</a> (or maybe <code>crun</code>? I can&rsquo;t keep track honestly) under the hood, so I thought &ndash;
well, maybe I can just use <code>runc</code> directly to make my own tool that starts
containers faster than Docker does!</p>

<p>I tried to do this 6 months ago and I don&rsquo;t remember most of the details, but basically
I spent 8 hours working on it, got frustrated because I couldn&rsquo;t get anything
to work, and gave up.</p>

<p>One specific detail I remember struggling with was setting up a working <code>/dev</code>
for my programs to use.</p>

<h3 id="enter-bubblewrap">enter bubblewrap</h3>

<p>Okay, that was a very long preamble so let&rsquo;s get to the point! Last week, I
discovered a tool called <code>bubblewrap</code> that was basically exactly the thing I
was trying to build with <code>runc</code> in my failed attempt, except that it actually
works and has many more features and it&rsquo;s built by people who know things about
security! Hooray!</p>

<p>The interface to bubblewrap is pretty different than the interface to Docker &ndash;
it&rsquo;s much lower level.  There&rsquo;s no concept of a container image &ndash; instead you
map a bunch of directories on your host to directories in the container.</p>

<p>For example, here&rsquo;s how to run a container with the same root directory as your
host operating system, but with only read access to that root directory, and only write access to <code>/tmp</code>.</p>

<pre><code>bwrap \
    --ro-bind / / \
    --bind /tmp /tmp \
    --proc /proc --dev /dev \
    --unshare-pid \
    --unshare-net \
    bash
</code></pre>

<p>For example, you could imagine running some untrusted process under bubblewrap
this way and then putting all the files you want the process to be able to access in <code>/tmp</code>.</p>

<h3 id="bubblewrap-runs-containers-as-an-unprivileged-non-root-user">bubblewrap runs containers as an unprivileged (non-root) user</h3>

<p>Like podman, bubblewrap runs containers as a non-root user, using user
namespaces. It can also run containers as root, but in this post we&rsquo;re just
going to be talking about using it as an unprivileged user.</p>

<h3 id="bubblewrap-is-fast">bubblewrap is fast</h3>

<p>Let&rsquo;s see how long it takes to run <code>ls</code> in a bubblewrap container!</p>

<pre><code>$ time bwrap --ro-bind / / --proc /proc --dev /dev --unshare-pid ls /
Executed in    8.04 millis
</code></pre>

<p>That&rsquo;s a big difference! 8ms is a lot faster than 279ms.</p>

<p>Of course, like we said before, the reason bubblewrap is faster is that it does
a lot less. So let&rsquo;s talk about some things bubblewrap doesn&rsquo;t do.</p>

<h3 id="some-things-bubblewrap-doesn-t-do">some things bubblewrap doesn&rsquo;t do</h3>

<p>Here are some things that Docker/podman do that bubblewrap doesn&rsquo;t do:</p>

<ul>
<li>set up overlayfs mounts for you, so that your changes to the filesystem don&rsquo;t affect the base image</li>
<li>set up networking bridges so that you can connect to a webserver inside the container</li>
<li>probably a bunch more stuff that I&rsquo;m not thinking of</li>
</ul>

<p>In general, bubblewrap is a much lower level tool than something like Docker.</p>

<p>Also, bubblewrap seems to have pretty different goals than Docker &ndash; the README
seems to say that it&rsquo;s intended as a tool for sandboxing desktop software (I
think it comes from <a href="https://flatpak.org/">flatpak</a>).</p>

<h3 id="running-a-container-image-with-bubblewrap">running a container image with bubblewrap</h3>

<p>I couldn&rsquo;t find instructions for running a Docker container image with
bubblewrap, so here they are. Basically I just use Docker to download the
container image and put it into a directory and then run it with <code>bwrap</code>:</p>

<p>There&rsquo;s also a tool called <a href="https://github.com/projectatomic/bwrap-oci">bwrap-oci</a> which looks cool but I
couldn&rsquo;t get it to compile.</p>

<pre><code>mkdir rootfs
docker export $(docker create frapsoft/fish) | tar -C rootfs -xf -
bwrap \
    --bind $PWD/rootfs / \
    --proc /proc --dev /dev \
    --uid 0 \
    --unshare-pid \
    --unshare-net \
    fish
</code></pre>

<p>One important thing to note is that this doesn&rsquo;t create a temporary overlay
filesystem for the container&rsquo;s file writes, so it&rsquo;ll let the container edit
files in the image.</p>

<p>I wrote a post about <a href="https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/">overlay filesystems</a> if
you want to see how you could do that yourself though.</p>

<h3 id="running-containers-with-bubblewrap-isn-t-the-same-as-with-podman">running &ldquo;containers&rdquo; with bubblewrap isn&rsquo;t the same as with podman</h3>

<p>I just gave an example of how to &ldquo;run a container&rdquo; with bubblewrap, and you
might think &ldquo;cool, this is just like podman but faster!&rdquo;. It is not, and it&rsquo;s
actually unlike using podman in even more ways than I expected.</p>

<p>I put &ldquo;container&rdquo; in scare quotes because there are two ways to define &ldquo;container&rdquo;:</p>

<ul>
<li>something that implements <a href="https://opencontainers.org/about/overview/">OCI runtime specification</a></li>
<li>any way of running a process in a way that&rsquo;s somehow isolated from the host system</li>
</ul>

<p>bubblewrap is a &ldquo;container&rdquo; tool in the second sense. It definitely provides
isolation, and it does that using the same features &ndash; Linux namespaces &ndash; as
Docker.</p>

<p>But it&rsquo;s not a container tool in the first sense. And it&rsquo;s a lower level tool
so you can get into a bunch of weird states and you really need to think about
all the weird details of how container work while using it.</p>

<p>For the rest of the post I&rsquo;m going to talk about some weird things that can
happen with bubblewrap that would not happen with podman/Docker.</p>

<h3 id="weird-thing-1-processes-that-don-t-exist">weird thing 1: processes that don&rsquo;t exist</h3>

<p>Here&rsquo;s an example of a weird situation I got into with bubblewrap that confused
me for a minute:</p>

<pre><code>$ bwrap --ro-bind / / --unshare-all bash
$ ps aux
... some processes
root      390073  0.0  0.0   2848   124 pts/9    S    14:28   0:00 bwrap --ro-bind / / --unshare-all --uid 0 bash
... some other processes
$ kill 390073
bash: kill: (390073) - No such process
$ ps aux | grep 390073
root      390073  0.0  0.0   2848   124 pts/9    S    14:28   0:00 bwrap --ro-bind / / --unshare-all --uid 0 bash
</code></pre>

<p>Here&rsquo;s what happened</p>

<ul>
<li>I started a bash shell inside bubblewrap</li>
<li>I ran <code>ps aux</code>, and saw a process with PID  <code>390073</code></li>
<li>I try to kill the process. It fails with the error <code>no such process</code>. What?</li>
<li>I ran <code>ps aux</code>, and still see the process with PID  <code>390073</code></li>
</ul>

<p>What&rsquo;s going on? Why doesn&rsquo;t the process <code>390073</code> exist, even though <code>ps</code> says it does? Isn&rsquo;t that impossible?</p>

<p>Well, the problem is that <code>ps</code> doesn&rsquo;t actually list all the processes in your
current PID namespace. Instead, it iterates through all the entries in <code>/proc</code>
and prints those out. Usually, what&rsquo;s in <code>/proc</code> is actually the same as the processes on your system.</p>

<p>But with Linux containers these things can get out of sync. What&rsquo;s happening in
this example is that we have the <code>/proc</code> from the host PID namespace, but those
aren&rsquo;t actually the processes that we have access to in our PID namespace.</p>

<p>Passing <code>--proc /proc</code> to bwrap fixes the issue &ndash; <code>ps</code> then actually lists the correct processes.</p>

<pre><code>$ bwrap --ro-bind / / --unshare-all --dev /dev --proc /proc ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bork           1  0.0  0.0   3644   136 ?        S+   16:21   0:00 bwrap --ro-bind / / --unshare-all --dev /dev --proc /proc ps au
bork           2  0.0  0.0  21324  1552 ?        R+   16:21   0:00 ps aux
</code></pre>

<p>Just 2 processes! Everything is normal!</p>

<h3 id="weird-thing-2-trying-to-listen-on-port-80">weird thing 2: trying to listen on port 80</h3>

<p>Passing <code>--uid 0</code> to bubblewrap makes the user inside the container <code>root</code>. You
might think that this means that the root user has administrative privileges
inside the container, but that&rsquo;s not true!</p>

<p>For example, let&rsquo;s try to listen on port 80:</p>

<pre><code>$ bwrap --ro-bind / / --unshare-all --uid 0 nc -l 80
nc: Permission denied
</code></pre>

<p>What&rsquo;s going on here is that the new root user actually doesn&rsquo;t have the
<strong>capabilities</strong> it needs to listen on port 80. (you need special permissions
to listen on ports less than 1024, and 80 is less than 1024)</p>

<p>There&rsquo;s actually a capability specifically for listening on privileged ports
called <code>CAP_NET_BIND_SERVICE</code>.</p>

<p>So to fix this all we need to do is to tell bubblewrap to give our user that
capability.</p>

<pre><code>$ bwrap --ro-bind / / --unshare-all --uid 0 --cap-add cap_net_bind_service nc -l 80
(no output, success!!!)
</code></pre>

<p>This works! Hooray!</p>

<h3 id="finding-the-right-capabilities-is-pretty-annoying">finding the right capabilities is pretty annoying</h3>

<p>bubblewrap doesn&rsquo;t give out any capabilities by default, and I find that
figuring out all the right capabilities and adding them manually is kind of
annoying. Basically my process is</p>

<ul>
<li>run the thing</li>
<li>see what fails</li>
<li>read <code>man capabilities</code> to figure out what capabilities I&rsquo;m missing</li>
<li>add the capability with <code>--cap-add</code></li>
<li>repeat until everything is running</li>
</ul>

<p>But that&rsquo;s the price I pay for wanting things to be fast I guess :)</p>

<h3 id="weird-thing-2b-dev-dev-makes-listening-on-privileged-ports-not-work">weird thing 2b:  <code>--dev /dev</code> makes listening on privileged ports not work</h3>

<p>One other strange thing is that if I take the exact same command above (which
worked!) and add <code>--dev /dev</code> (to set up the <code>/dev/</code> directory), it causes it to not work again:</p>

<pre><code>$ bwrap --ro-bind / / --dev /dev --unshare-all --uid 0 --cap-add cap_net_bind_service nc -l 80
nc: Permission denied
</code></pre>

<p>I think this might be a bug in bubblewrap, but I haven&rsquo;t mustered the courage
to dive into the bubblewrap code and start investigating yet. Or maybe there&rsquo;s
something obvious I&rsquo;m missing!</p>

<h3 id="weird-thing-3-uid-mappings">weird thing 3: UID mappings</h3>

<p>Another slightly weird thing was &ndash; I tried to run <code>apt-get update</code> inside a bubblewrap Ubuntu container and everything went very poorly.</p>

<p>Here&rsquo;s how I ran <code>apt-get update</code> inside the Ubuntu container:</p>

<pre><code>mkdir rootfs
docker export $(docker create ubuntu:20.04) | tar -C rootfs -xf -
bwrap \
    --bind $PWD/rootfs / \
    --proc /proc\
    --uid 0 \
    --unshare-pid \
    apt-get update
</code></pre>

<p>And here are the error messages:</p>

<pre><code>E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 100 failed - seteuid (22: Invalid argument)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
.... lots more similar errors
</code></pre>

<p>At first I thought &ldquo;ok, this is a capabilities problem, I need to set
<code>CAP_SETGID</code> or something to give the container permission to change groups. But I did that and it didn&rsquo;t help at all!</p>

<p>I think what&rsquo;s going on here is a problem with UID maps. What are UID maps?
Well, every time you run a container using &ldquo;user namespaces&rdquo; (which podman is
doing), it creates a mapping of UIDs inside the container to UIDs on the host.</p>

<p>Let&rsquo;s look that the UID maps! Here&rsquo;s how to do that:</p>

<pre><code>root@kiwi:/# cat /proc/self/uid_map 
         0       1000          1
root@kiwi:/# cat /proc/self/gid_map 
      1000       1000          1
</code></pre>

<p>This is saying that user 0 in the container is mapped to user 1000 on in the
host, and group 1000 is mapped to group 1000. (My normal user&rsquo;s UID/GID is 1000, so this makes sense). You can find out
about this <code>uid_map</code> file in <code>man user_namespaces</code>.</p>

<p>All other users/groups that aren&rsquo;t 1000 are mapped to user 65534 by default, according
to <code>man user_namespaces</code>.</p>

<h3 id="what-s-going-on-non-mapped-users-can-t-be-used">what&rsquo;s going on: non-mapped users can&rsquo;t be used</h3>

<p>The only users and groups that have been mapped are <code>0</code> and <code>1000</code>. But <code>man user_namespaces</code> says:</p>

<blockquote>
<p>After the uid_map and gid_map files have been written, only the mapped values may be used in system calls that change user and group IDs.</p>
</blockquote>

<p><code>apt</code> is trying to use users 100 and 65534. Those aren&rsquo;t on the list of mapped
users! So they can&rsquo;t be used!</p>

<p>This works fine in podman, because podman sets up its UID and GID mappings differently:</p>

<pre><code>$ podman run -it ubuntu:20.04 bash
root@793d03a4d773:/# cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
root@793d03a4d773:/# cat /proc/self/gid_map
         0       1000          1
         1     100000      65536
</code></pre>

<p>All the users get mapped, not just 1000.</p>

<p>I don&rsquo;t quite know how to fix this, but I think it&rsquo;s probably possible in
bubblewrap to set up the uid mappings the same way as podman does &ndash; there&rsquo;s an
<a href="https://github.com/containers/bubblewrap/issues/468">issue about it here that links to a workaround</a>.</p>

<p>But this wasn&rsquo;t an actual problem I was trying to solve so I didn&rsquo;t dig further
into it.</p>

<h3 id="a-quick-note-on-firecracker">a quick note on Firecracker</h3>

<p>Someone asked &ldquo;would Firecracker work here?&rdquo; (I <a href="https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/">wrote about Firecracker</a> last year).</p>

<p>My experience with Firecracker VMs is that they use kind of a lot of RAM (like
50MB?), which makes sense because they&rsquo;re VMs. And when I tried Firecracker on
a tiny machine (with ~256MB of RAM / a tiny CPU), the startup times were 2-3
seconds.</p>

<p>I&rsquo;m sure it&rsquo;s possible to optimize Firecracker to be a bit faster, but at the
end of the day I think it&rsquo;s a VM and it&rsquo;s not going to be anywhere near as low
overhead as a process &ndash; there&rsquo;s a whole operating system to start!</p>

<p>So Firecracker would add a lot more overhead than I want in this case.</p>

<h3 id="bubblewrap-works-pretty-great">bubblewrap works pretty great!</h3>

<p>I&rsquo;ve talked about a bunch of issues, but the things I&rsquo;ve been trying to do in bubblewrap
have been very constrained and it&rsquo;s actually been pretty simple. For example, I
was working on a git project where I really just want to run <code>git</code> inside a
container and map a git repository from the host.</p>

<p>That&rsquo;s very simple to get to work with bubblewrap! There were basically no weird problems!
It&rsquo;s really fast!</p>

<p>So I&rsquo;m pretty excited about this tool and I might use it for more stuff in the
future.</p>


guid: https://jvns.ca/blog/2022/07/09/monitoring-small-web-services/
title: Monitoring tiny web services
url: https://jvns.ca/blog/2022/07/09/monitoring-small-web-services/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! I&rsquo;ve started to run a few more servers recently
(<a href="https://nginx-playground.wizardzines.com">nginx playground</a>,
<a href="https://messwithdns.net">mess with dns</a>,
<a href="https://dns-lookup.jvns.ca">dns lookup</a>), so I&rsquo;ve been
thinking about monitoring.</p>

<p>It wasn&rsquo;t initially totally obvious to me how to monitor these websites, so I
wanted to quickly write up what how I did it.</p>

<p>I&rsquo;m not going to talk about how to monitor Big Serious Mission Critical
websites at all, only tiny unimportant websites.</p>

<h3 id="goal-spend-approximately-0-time-on-operations">goal: spend approximately 0 time on operations</h3>

<p>I want the sites to mostly work, but I also want to spend approximately 0% of
my time on the ongoing operations.</p>

<p>I was initially very wary of running servers at all because at my last job I
was on a <sup>24</sup>&frasl;<sub>7</sub> oncall rotation for some critical services, and in my mind &ldquo;being
responsible for servers&rdquo; meant &ldquo;get woken up at 2am to fix the servers&rdquo; and
&ldquo;have lots of complicated dashboards&rdquo;.</p>

<p>So for a while I only made static websites so that I wouldn&rsquo;t have to think
about servers.</p>

<p>But eventually I realized that any server I was going to write was going to be
very low stakes, if they occasionally go down for 2 hours it&rsquo;s no big deal, and
I could just set up some very simple monitoring to help keep them running.</p>

<h3 id="not-having-monitoring-sucks">not having monitoring sucks</h3>

<p>At first I didn&rsquo;t set up any monitoring for my servers at all. This had the
extremely predictable outcome of &ndash; sometimes the site broke, and I didn&rsquo;t find
out about it until somebody told me!</p>

<h3 id="step-1-an-uptime-checker">step 1: an uptime checker</h3>

<p>The first step was to set up an uptime checker. There are tons of these out
there, the ones I&rsquo;m using right now are <a href="https://updown.io/">updown.io</a> and
<a href="https://uptimerobot.com/">uptime robot</a>. I like updown&rsquo;s user interface and
<a href="https://updown.io/#pricing">pricing</a> structure more (it&rsquo;s per request instead of a monthly fee), but uptime
robot has a more generous free tier.</p>

<p>These</p>

<ol>
<li>check that the site is up</li>
<li>if it goes down, it emails me</li>
</ol>

<p>I find that email notifications are a good level for me, I&rsquo;ll find out pretty
quickly if the site goes down but it doesn&rsquo;t wake me up or anything.</p>

<h3 id="step-2-an-end-to-end-healthcheck">step 2: an end-to-end healthcheck</h3>

<p>Next, let&rsquo;s talk about what &ldquo;check that the site is up&rdquo; actually means.</p>

<p>At first I just made one of my healthcheck endpoints a function that returned
<code>200 OK</code> no matter what.</p>

<p>This is kind of useful &ndash; it told me that the server was on!</p>

<p>But unsurprisingly I ran into problems because it wasn&rsquo;t checking that the API
was actually <em>working</em> &ndash; sometimes the healthcheck succeeded even though the
rest of the service had actually gotten into a bad state.</p>

<p>So I updated it to actually make a real API request and make sure it
succeeded.</p>

<p>All of my services do very few things (the nginx playground has just 1
endpoint), so it&rsquo;s pretty easy to set up a healthcheck that actually runs
through most of the actions the service is supposed to do.</p>

<p>Here&rsquo;s what the end-to-end healthcheck handler for the nginx playground looks
like. It&rsquo;s very basic: it just makes another POST request (to itself) and
checks if that request succeeds or fails.</p>

<pre><code>func healthHandler(w http.ResponseWriter, r *http.Request) {
	// make a request to localhost:8080 with `healthcheckJSON` as the body
	// if it works, return 200
	// if it doesn't, return 500
	client := http.Client{}
	resp, err := client.Post(&quot;http://localhost:8080/&quot;, &quot;application/json&quot;, strings.NewReader(healthcheckJSON))
	if err != nil {
		log.Println(err)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
	if resp.StatusCode != http.StatusOK {
		log.Println(resp.StatusCode)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
	w.WriteHeader(http.StatusOK)
}
</code></pre>

<h3 id="healthcheck-frequency-hourly">healthcheck frequency: hourly</h3>

<p>Right now I&rsquo;m running most of my healthchecks every hour, and some every 30
minutes.</p>

<p>I run them hourly because updown.io&rsquo;s pricing is per healthcheck, I&rsquo;m
monitoring 18 different URLs, and I wanted to keep my healthcheck budget pretty
minimal at $5/year.</p>

<p>Taking an hour to find out that one of these websites has gone down seems ok to
me &ndash; if there is a problem there&rsquo;s no guarantee I&rsquo;ll get to fixing it all that
quickly anyway.</p>

<p>If it were free to run them more often I&rsquo;d probably run them every 5-10 minutes instead.</p>

<h3 id="step-3-automatically-restart-if-the-healthcheck-fails">step 3: automatically restart if the healthcheck fails</h3>

<p>Some of my websites are on fly.io, and fly has a pretty standard feature where
I can configure a HTTP healthcheck for a service and restart the service if the
healthcheck starts failing.</p>

<p>&ldquo;Restart a lot&rdquo; is a very useful strategy to paper over bugs that I haven&rsquo;t
gotten around to fixing yet &ndash; for a while the nginx playground had a process
leak where <code>nginx</code> processes weren&rsquo;t getting terminated, so the server kept
running out of RAM.</p>

<p>With the healthcheck, the result of this was that every day or so, this would happen:</p>

<ul>
<li>the server ran out of RAM</li>
<li>the healthcheck started failing</li>
<li>it get restarted</li>
<li>everything was fine again</li>
<li>repeat the whole saga again some number of hours later</li>
</ul>

<p>Eventually I got around to actually fixing the process leak, but it was nice to
have a workaround in place that could keep things running while I was
procrastinating fixing the bug.</p>

<p>These healthchecks to decide whether to restart the service run more often: every 5 minutes or so.</p>

<h3 id="this-is-not-the-best-way-to-monitor-big-services">this is not the best way to monitor Big Services</h3>

<p>This is probably obvious and I said this already at the beginning, but &ldquo;write
one HTTP healthcheck&rdquo; is not the best approach for monitoring a large complex
service. But I won&rsquo;t go into that because that&rsquo;s not what this post is about.</p>

<h3 id="it-s-been-working-well-so-far">it&rsquo;s been working well so far!</h3>

<p>I originally wrote this post 3 months ago in April, but I waited until now to
publish it to make sure that the whole setup was working.</p>

<p>It&rsquo;s made a pretty big difference &ndash; before I was having some very silly
downtime problems, and now for the last few months the sites have been up
99.95% of the time!</p>


guid: https://jvns.ca/blog/2022/07/20/pseudoterminals/
title: What happens when you press a key in your terminal?
url: https://jvns.ca/blog/2022/07/20/pseudoterminals/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>I&rsquo;ve been confused about what&rsquo;s going on with terminals for a long time.</p>

<p>But this past week I was using <a href="https://xtermjs.org/">xterm.js</a> to display an
interactive terminal in a browser and I finally thought to ask a pretty basic
question: when you press a key on your keyboard in a terminal (like <code>Delete</code>, or <code>Escape</code>, or <code>a</code>), which
bytes get sent?</p>

<p>As usual we&rsquo;ll answer that question by doing some experiments and seeing what happens :)</p>

<h3 id="remote-terminals-are-very-old-technology">remote terminals are very old technology</h3>

<p>First, I want to say that displaying a terminal in the browser with <code>xterm.js</code>
might seem like a New Thing, but it&rsquo;s really not. In the 70s, computers were
expensive. So many employees at an institution would share a single computer,
and each person could have their own &ldquo;terminal&rdquo; to that computer.</p>

<p>For example, here&rsquo;s a photo of a VT100 terminal from the 70s or 80s. This looks like
it could be a computer (it&rsquo;s kind of big!), but it&rsquo;s not &ndash; it just displays
whatever information the actual computer sends it.</p>

<p><a title="Jason Scott, CC BY 2.0 &lt;https://creativecommons.org/licenses/by/2.0&gt;, via Wikimedia Commons" href="https://commons.wikimedia.org/wiki/File:DEC_VT100_terminal.jpg"><img width="512" alt="DEC VT100 terminal" src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/DEC_VT100_terminal.jpg/512px-DEC_VT100_terminal.jpg"></a></p>

<p>Of course, in the 70s they didn&rsquo;t use websockets for this, but the information
being sent back and forth is more or less the same as it was then.</p>

<p><small>
(the terminal in that photo is from the <a href="https://livingcomputers.org/">Living Computer Museum</a> in Seattle which I got to visit once and write FizzBuzz in <code>ed</code> on a very old Unix system, so it&rsquo;s possible that I&rsquo;ve actually used that machine or one of its siblings! I really hope the Living Computer Museum opens again, it&rsquo;s very cool to get to play with old computers.)
</small></p>

<h3 id="what-information-gets-sent">what information gets sent?</h3>

<p>It&rsquo;s obvious that if you want to connect to a remote computer (with <code>ssh</code> or
using <code>xterm.js</code> and a websocket, or anything else), then some information
needs to be sent between the client and the server.</p>

<p>Specifically:</p>

<ul>
<li>the <strong>client</strong> needs to send the keystrokes that the user typed in (like <code>ls -l</code>)</li>
<li>the <strong>server</strong> needs to tell the client what to display on the screen</li>
</ul>

<p>Let&rsquo;s look at a real program that&rsquo;s running a remote terminal in a browser and see what information gets sent back and forth!</p>

<h3 id="we-ll-use-goterm-to-experiment">we&rsquo;ll use <code>goterm</code> to experiment</h3>

<p>I found this tiny program on GitHub called
<a href="https://github.com/freman/goterm">goterm</a> that runs a Go server that lets you
interact with a terminal in the browser using <code>xterm.js</code>. This program is very insecure but it&rsquo;s simple and great for learning.</p>

<p>I <a href="https://github.com/jvns/goterm">forked it</a> to make it work with the latest xterm.js,
since it was last updated 6 years ago. Then I added some logging statements to
print out every time bytes are sent/received over the websocket.</p>

<p>Let&rsquo;s look at sent and received during a few different terminal interactions!</p>

<h3 id="example-ls">example: <code>ls</code></h3>

<p>First, let&rsquo;s run <code>ls</code>. Here&rsquo;s what I see on the <code>xterm.js</code> terminal:</p>

<pre><code>bork@kiwi:/play$ ls
file
bork@kiwi:/play$
</code></pre>

<p>and here&rsquo;s what gets sent and received: (in my code, I log <code>sent: [bytes]</code> every time the client sends bytes and <code>recv: [bytes]</code> every time it receives bytes from the server)</p>

<pre><code>sent: &quot;l&quot;
recv: &quot;l&quot;
sent: &quot;s&quot;
recv: &quot;s&quot;
sent: &quot;\r&quot;
recv: &quot;\r\n\x1b[?2004l\r&quot;
recv: &quot;file\r\n&quot;
recv: &quot;\x1b[?2004hbork@kiwi:/play$ &quot;
</code></pre>

<p>I noticed 3 things in this output:</p>

<ol>
<li>Echoing: The client sends <code>l</code> and then immediately receives an <code>l</code> sent
back. I guess the idea here is that the client is really dumb &ndash; it doesn&rsquo;t
know that when I type an <code>l</code>, I want an <code>l</code> to be echoed back to the screen.
It has to be told explicitly by the server process to display it.</li>
<li>The newline: when I press enter, it sends a <code>\r</code> (carriage return) symbol and not a <code>\n</code> (newline)</li>
<li>Escape sequences: <code>\x1b</code> is the ASCII escape character, so <code>\x1b[?2004h</code> is
telling the terminal to display something or other. I think this is a colour
sequence but I&rsquo;m not sure. We&rsquo;ll talk a little more about escape sequences later.</li>
</ol>

<p>Okay, now let&rsquo;s do something slightly more complicated.</p>

<h3 id="example-ctrl-c">example: <code>Ctrl+C</code></h3>

<p>Next, let&rsquo;s see what happens when we interrupt a process with <code>Ctrl+C</code>. Here&rsquo;s what I see in my terminal:</p>

<pre><code>bork@kiwi:/play$ cat
^C
bork@kiwi:/play$
</code></pre>

<p>And here&rsquo;s what the client sends and receives.</p>

<pre><code>sent: &quot;c&quot;
recv: &quot;c&quot;
sent: &quot;a&quot;
recv: &quot;a&quot;
sent: &quot;t&quot;
recv: &quot;t&quot;
sent: &quot;\r&quot;
recv: &quot;\r\n\x1b[?2004l\r&quot;
sent: &quot;\x03&quot;
recv: &quot;^C&quot;
recv: &quot;\r\n&quot;
recv: &quot;\x1b[?2004h&quot;
recv: &quot;bork@kiwi:/play$ &quot;
</code></pre>

<p>When I press <code>Ctrl+C</code>, the client sends <code>\x03</code>. If I look up an ASCII table,
<code>\x03</code> is &ldquo;End of Text&rdquo;, which seems reasonable. I thought this was really cool
because I&rsquo;ve always been a bit confused about how Ctrl+C works &ndash; it&rsquo;s good to
know that it&rsquo;s just sending an <code>\x03</code> character.</p>

<p>I believe the reason <code>cat</code> gets interrupted when we press <code>Ctrl+C</code> is that the
Linux kernel on the server side receives this <code>\x03</code> character, recognizes that
it means &ldquo;interrupt&rdquo;, and then sends a <code>SIGINT</code> to the process that owns the
pseudoterminal&rsquo;s process group. So it&rsquo;s handled in the kernel and not in
userspace.</p>

<h3 id="example-ctrl-d">example: <code>Ctrl+D</code></h3>

<p>Let&rsquo;s try the exact same thing, except with <code>Ctrl+D</code>. Here&rsquo;s what I see in my terminal:</p>

<pre><code>bork@kiwi:/play$ cat
bork@kiwi:/play$
</code></pre>

<p>And here&rsquo;s what gets sent and received:</p>

<pre><code>sent: &quot;c&quot;
recv: &quot;c&quot;
sent: &quot;a&quot;
recv: &quot;a&quot;
sent: &quot;t&quot;
recv: &quot;t&quot;
sent: &quot;\r&quot;
recv: &quot;\r\n\x1b[?2004l\r&quot;
sent: &quot;\x04&quot;
recv: &quot;\x1b[?2004h&quot;
recv: &quot;bork@kiwi:/play$ &quot;
</code></pre>

<p>It&rsquo;s very similar to <code>Ctrl+C</code>, except that <code>\x04</code> gets sent instead of <code>\x03</code>.
Cool! <code>\x04</code> corresponds to ASCII &ldquo;End of Transmission&rdquo;.</p>

<h3 id="what-about-ctrl-another-letter">what about Ctrl + another letter?</h3>

<p>Next I got curious about &ndash; if I send <code>Ctrl+e</code>, what byte gets sent?</p>

<p>It turns out that it&rsquo;s literally just the number of that letter in the alphabet, like this:</p>

<ul>
<li><code>Ctrl+a</code> =&gt; 1</li>
<li><code>Ctrl+b</code> =&gt; 2</li>
<li><code>Ctrl+c</code> =&gt; 3</li>
<li><code>Ctrl+d</code> =&gt; 4</li>
<li>&hellip;</li>
<li><code>Ctrl+z</code> =&gt; 26</li>
</ul>

<p>Also, <code>Ctrl+Shift+b</code> does the exact same thing as <code>Ctrl+b</code> (it writes <code>0x2</code>).</p>

<p>What about other keys on the keyboard? Here&rsquo;s what they map to:</p>

<ul>
<li>Tab -&gt; 0x9 (same as Ctrl+I, since I is the 9th letter)</li>
<li>Escape -&gt; <code>\x1b</code><br /></li>
<li>Backspace -&gt; <code>\x7f</code></li>
<li>Home -&gt; <code>\x1b[H</code></li>
<li>End: <code>\x1b[F</code></li>
<li>Print Screen: <code>\x1b\x5b\x31\x3b\x35\x41</code></li>
<li>Insert: <code>\x1b\x5b\x32\x7e</code></li>
<li>Delete -&gt; <code>\x1b\x5b\x33\x7e</code></li>
<li>My <code>Meta</code> key does nothing at all</li>
</ul>

<p>What about Alt? From my experimenting (and some Googling), it seems like <code>Alt</code>
is literally the same as &ldquo;Escape&rdquo;, except that pressing <code>Alt</code> by itself doesn&rsquo;t
send any characters to the terminal and pressing <code>Escape</code> by itself does. So:</p>

<ul>
<li>alt + d =&gt; <code>\x1bd</code> (and the same for every other letter)</li>
<li>alt + shift + d =&gt; <code>\x1bD</code> (and the same for every other letter)</li>
<li>etcetera</li>
</ul>

<p>Let&rsquo;s look at one more example!</p>

<h3 id="example-nano">example: <code>nano</code></h3>

<p>Here&rsquo;s what gets sent and received when I run the text editor <code>nano</code>:</p>

<pre><code>recv: &quot;\r\x1b[Kbork@kiwi:/play$ &quot;
sent: &quot;n&quot; [[]byte{0x6e}]
recv: &quot;n&quot;
sent: &quot;a&quot; [[]byte{0x61}]
recv: &quot;a&quot;
sent: &quot;n&quot; [[]byte{0x6e}]
recv: &quot;n&quot;
sent: &quot;o&quot; [[]byte{0x6f}]
recv: &quot;o&quot;
sent: &quot;\r&quot; [[]byte{0xd}]
recv: &quot;\r\n\x1b[?2004l\r&quot;
recv: &quot;\x1b[?2004h&quot;
recv: &quot;\x1b[?1049h\x1b[22;0;0t\x1b[1;16r\x1b(B\x1b[m\x1b[4l\x1b[?7h\x1b[39;49m\x1b[?1h\x1b=\x1b[?1h\x1b=\x1b[?25l&quot;
recv: &quot;\x1b[39;49m\x1b(B\x1b[m\x1b[H\x1b[2J&quot;
recv: &quot;\x1b(B\x1b[0;7m  GNU nano 6.2 \x1b[44bNew Buffer \x1b[53b \x1b[1;123H\x1b(B\x1b[m\x1b[14;38H\x1b(B\x1b[0;7m[ Welcome to nano.  For basic help, type Ctrl+G. ]\x1b(B\x1b[m\r\x1b[15d\x1b(B\x1b[0;7m^G\x1b(B\x1b[m Help\x1b[15;16H\x1b(B\x1b[0;7m^O\x1b(B\x1b[m Write Out   \x1b(B\x1b[0;7m^W\x1b(B\x1b[m Where Is    \x1b(B\x1b[0;7m^K\x1b(B\x1b[m Cut\x1b[15;61H&quot;
</code></pre>

<p>You can see some text from the UI in there like &ldquo;GNU nano 6.2&rdquo;, and these
<code>\x1b[27m</code> things are escape sequences. Let&rsquo;s talk about escape sequences a bit!</p>

<h3 id="ansi-escape-sequences">ANSI escape sequences</h3>

<p>These <code>\x1b[</code> things above that <code>nano</code> is sending the client are called &ldquo;escape sequences&rdquo; or &ldquo;escape codes&rdquo;.
This is because they all start with <code>\x1b</code>, the &ldquo;escape&rdquo; character. .  They change the
cursor&rsquo;s position, make text bold or underlined, change colours, etc. <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">Wikipedia has some history</a> if you&rsquo;re interested.</p>

<p>As a simple example: if you run</p>

<pre><code>echo -e '\e[0;31mhi\e[0m there'
</code></pre>

<p>in your terminal, it&rsquo;ll print out &ldquo;hi there&rdquo; where &ldquo;hi&rdquo; is in red and &ldquo;there&rdquo;
is in black. <a href="https://misc.flogisoft.com/bash/tip_colors_and_formatting">This page</a> has some nice
examples of escape codes for colors and formatting.</p>

<p>I think there are a few different standards for escape codes, but my
understanding is that the most common set of escape codes that people use on
Unix come from the VT100 (that old terminal in the picture at the top of the
blog post), and hasn&rsquo;t really changed much in the last 40 years.</p>

<p>Escape codes are why your terminal can get messed up if you <code>cat</code> a bunch of binary to
your screen &ndash; usually you&rsquo;ll end up accidentally printing a bunch of random
escape codes which will mess up your terminal &ndash; there&rsquo;s bound to be a <code>0x1b</code>
byte in there somewhere if you <code>cat</code> enough binary to your terminal.</p>

<h3 id="can-you-type-in-escape-sequences-manually">can you type in escape sequences manually?</h3>

<p>A few sections back, we talked about how the <code>Home</code> key maps to <code>\x1b[H</code>. Those 3 bytes are <code>Escape + [ + H</code> (because Escape is
<code>\x1b</code>).</p>

<p>And if I manually type Escape, then [, then H in the
<code>xterm.js</code> terminal, I end up at the beginning of the line, exactly the same as if I&rsquo;d pressed <code>Home</code>.</p>

<p>I noticed that this didn&rsquo;t work in <code>fish</code> on my computer though &ndash; if I typed
<code>Escape</code> and then <code>[</code>, it just printed out <code>[</code> instead of letting me continue the
escape sequence. I asked my friend Jesse who has written <a href="https://github.com/doy/vt100-rust">a bunch of Rust
terminal code</a> about this and Jesse told me
that a lot of programs implement a <strong>timeout</strong> for escape codes &ndash; if you don&rsquo;t
press another key after some minimum amount of time, it&rsquo;ll decide that it&rsquo;s
actually not an escape code anymore.</p>

<p>Apparently this is configurable in fish with <code>fish_escape_delay_ms</code>, so I ran
<code>set fish_escape_delay_ms 1000</code> and then I was able to type in escape codes by
hand. Cool!</p>

<h3 id="terminal-encoding-is-kind-of-weird">terminal encoding is kind of weird</h3>

<p>I want to pause here for a minute here and say that the way the keys you get
pressed get mapped to bytes is pretty weird. Like, if we were designing
the way keys are encoded from scratch today, we would probably not set it up so
that:</p>

<ul>
<li><code>Ctrl + a</code> does the exact same thing as <code>Ctrl + Shift + a</code></li>
<li><code>Alt</code> is the same as <code>Escape</code></li>
<li>control sequences (like colours / moving the cursor around) use the same byte
as the <code>Escape</code> key, so that you need to rely on timing to determine if it
was a control sequence of the user just meant to press <code>Escape</code></li>
</ul>

<p>But all of this was designed in the 70s or 80s or something and then needed
to stay the same forever for backwards compatibility, so that&rsquo;s what we get :)</p>

<h3 id="changing-window-size">changing window size</h3>

<p>Not everything you can do in a terminal happens via sending bytes back and
forth. For example, when the terminal gets resized, we have to tell Linux that the window size has
changed in a different way.</p>

<p>Here&rsquo;s what the Go code in
<a href="https://github.com/freman/goterm/blob/a644c10e180ce8af789ea3e4e4892dcf078e97e2/main.go#L110-L115">goterm</a>
to do that looks like:</p>

<pre><code>syscall.Syscall(
    syscall.SYS_IOCTL,
    tty.Fd(),
    syscall.TIOCSWINSZ,
    uintptr(unsafe.Pointer(&amp;resizeMessage)),
)
</code></pre>

<p>This is using the <code>ioctl</code> system call. My understanding of <code>ioctl</code> is that it&rsquo;s
a system call for a bunch of random stuff that isn&rsquo;t covered by other system
calls, generally related to IO I guess.</p>

<p><code>syscall.TIOCSWINSZ</code> is an integer constant which which tells <code>ioctl</code> which
particular thing we want it to to in this case (change the window size of a
terminal).</p>

<h3 id="this-is-also-how-xterm-works">this is also how xterm works</h3>

<p>In this post we&rsquo;ve been talking about remote terminals, where the client and
the server are on different computers. But actually if you use a terminal
emulator like <code>xterm</code>, all of this works the exact same way, it&rsquo;s just harder
to notice because the bytes aren&rsquo;t being sent over a network connection.</p>

<h3 id="that-s-all-for-now">that&rsquo;s all for now!</h3>

<p>There&rsquo;s defimitely a lot more to know about terminals (we could talk more about
colours, or raw vs cooked mode, or unicode support, or the Linux pseudoterminal
interface) but I&rsquo;ll stop here because it&rsquo;s 10pm, this is getting kind of long,
and I think my brain cannot handle more new information about terminals today.</p>

<p><small>
Thanks to <a href="https://github.com/doy/">Jesse Luehrs</a> for answering a billion of my questions about terminals, all the mistakes are mine :)
</small></p>


guid: https://jvns.ca/blog/2022/07/28/toy-remote-login-server/
title: A toy remote login server
url: https://jvns.ca/blog/2022/07/28/toy-remote-login-server/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! The other day we talked about <a href="https://jvns.ca/blog/2022/07/20/pseudoterminals/">what happened when you press a key in your terminal</a>.</p>

<p>As a followup, I thought it might be fun to implement a program that&rsquo;s like a
tiny ssh server, but without the security. You can find it <a href="https://github.com/jvns/tiny-remote-login/blob/main/server.go">on github here</a>, and I&rsquo;ll explain how it works in this blog post.</p>

<h3 id="the-goal-ssh-to-a-remote-computer">the goal: &ldquo;ssh&rdquo; to a remote computer</h3>

<p>Our goal is to be able to login to a remote computer and run commands, like you
do with SSH or telnet.</p>

<p>The biggest difference between this program and SSH is that there&rsquo;s literally
no security (not even a password) &ndash; anyone who can make a TCP connection to
the server can get a shell and run commands.</p>

<p>Obviously this is not a useful program in real life, but our goal is to learn a
little more about how terminals works, not to write a useful program.</p>

<p>(I will run a version of it on the public internet for the next week though,
you can see how to connect to it at the end of this blog post)</p>

<h3 id="let-s-start-with-the-server">let&rsquo;s start with the server!</h3>

<p>We&rsquo;re also going to write a client, but the server is the interesting part, so
let&rsquo;s start there. We&rsquo;re going to write a server that listens on a TCP port (I
picked 7777) and creates remote terminals for any client that connects to it to
use.</p>

<p>When the server receives a new connection it needs to:</p>

<ol>
<li>create a pseudoterminal for the client to use</li>
<li>start a <code>bash</code> shell process for the client to use</li>
<li>connect <code>bash</code> to the pseudoterminal</li>
<li>continuously copy information back and forth between the TCP connection and
the pseudoterminal</li>
</ol>

<p>I just said the word &ldquo;pseudoterminal&rdquo; a lot, so let&rsquo;s talk about what that
means.</p>

<h3 id="what-s-a-pseudoterminal">what&rsquo;s a pseudoterminal?</h3>

<p>Okay, what the heck is a pseudoterminal?</p>

<p>A pseudoterminal is a lot like a bidirectional pipe or a socket &ndash; you have two
ends, and they can both send and receive information. You can read more about
the information being sent and received in <a href="https://jvns.ca/blog/2022/07/20/pseudoterminals/">what happens if you press a key in your terminal</a></p>

<p>Basically the idea is that on one end, we have a TCP connection, and on the
other end, we have a <code>bash</code> shell. So we need to hook one part of the
pseudoterminal up to the TCP connection and the other end to bash.</p>

<p>The two parts of the pseudoterminal are called:</p>

<ul>
<li>the &ldquo;pseudoterminal master&rdquo;. This is the end we&rsquo;re going to hook up to the TCP connection.</li>
<li>the &ldquo;slave pseudoterminal device&rdquo;. We&rsquo;re going to set our bash shell&rsquo;s <code>stdout</code>, <code>stderr</code>, and <code>stdin</code> to this.</li>
</ul>

<p>Once they&rsquo;re conected, we can communicate with <code>bash</code> over our TCP connection
and we&rsquo;ll have a remote shell!</p>

<h3 id="why-do-we-need-this-pseudoterminal-thing-anyway">why do we need this &ldquo;pseudoterminal&rdquo; thing anyway?</h3>

<p>You might be wondering &ndash; Julia, if a pseudoterminal is kind of like a socket,
why can&rsquo;t we just set our bash shell&rsquo;s <code>stdout</code> / <code>stderr</code> / <code>stdin</code> to the TCP
socket?</p>

<p>And you can! We could write a TCP connection handler like this that does exactly that, it&rsquo;s not a lot of code (<a href="https://github.com/jvns/tiny-remote-login/blob/main/server-notty.go">server-notty.go</a>).</p>

<pre><code>
func handle(conn net.Conn) {
	tty, _ := conn.(*net.TCPConn).File()
	// start bash with tcp connection as stdin/stdout/stderr
	cmd := exec.Command(&quot;bash&quot;)
	cmd.Stdin = tty
	cmd.Stdout = tty
	cmd.Stderr = tty
	cmd.Start()
}

</code></pre>

<p>It even kind of works &ndash; if we connect to it with <code>nc localhost 7778</code>, we can
run commands and look at their output.</p>

<p>But there are a few problems. I&rsquo;m not going to list all of them, just two.</p>

<p><strong>problem 1: Ctrl + C doesn&rsquo;t work</strong></p>

<p>The way Ctrl + C works in a remote login session is</p>

<ul>
<li>you press ctrl + c</li>
<li>That gets translated to <code>0x03</code> and sent through the TCP connection</li>
<li>The terminal receives it</li>
<li>the Linux kernel on the other end notes &ldquo;hey, that was a Ctrl + C!&rdquo;</li>
<li>Linux sends a <code>SIGINT</code> to the appropriate process (more on what the &ldquo;appropriate process&rdquo; is exactly later)</li>
</ul>

<p>If the &ldquo;terminal&rdquo; is just a TCP connection, this doesn&rsquo;t work, because when you
send <code>0x04</code> to a TCP connection, Linux won&rsquo;t magically send <code>SIGINT</code> to any
process.</p>

<p><strong>problem 2: <code>top</code> doesn&rsquo;t work</strong></p>

<p>When I try to run <code>top</code> in this shell, I get the error message <code>top: failed tty get</code>. If we strace it, we see this system call:</p>

<pre><code>ioctl(2, TCGETS, 0x7ffec4e68d60)        = -1 ENOTTY (Inappropriate ioctl for device)
</code></pre>

<p>So <code>top</code> is running an <code>ioctl</code> on its output file descriptor (2) to get some
information about the terminal. But Linux is like &ldquo;hey, this isn&rsquo;t a terminal!&rdquo;
and returns an error.</p>

<p>There are a bunch of other things that go wrong, but hopefully at this point
you&rsquo;re convinced that we actually need to set bash&rsquo;s stdout/stderr to be a
terminal, not some other thing like a socket.</p>

<p>So let&rsquo;s start looking at the server code and see what creating a
pseudoterminal actually looks like.</p>

<h3 id="step-1-create-a-pseudoterminal">step 1: create a pseudoterminal</h3>

<p>Here&rsquo;s some Go code to create a pseudoterminal on Linux. This is copied from <a href="https://github.com/creack/pty/blob/7de28cee0d53510e719c1aeb1850af0fa647c343/pty_linux.go">github.com/creack/pty</a>,
but I removed some of the error handling to make the logic a bit easier to follow:</p>

<pre><code>pty, _ := os.OpenFile(&quot;/dev/ptmx&quot;, os.O_RDWR, 0)
sname := ptsname(p)
unlockpt(p)
tty, _ := os.OpenFile(sname, os.O_RDWR|syscall.O_NOCTTY, 0)
</code></pre>

<p>In English, what we&rsquo;re doing is:</p>

<ul>
<li>open <code>/dev/ptmx</code> to get the &ldquo;pseudoterminal master&rdquo; Again, that&rsquo;s the part we&rsquo;re going to hook up to the TCP connection</li>
<li>get the filename of the &ldquo;slave pseudoterminal device&rdquo;, which is going to be <code>/dev/pts/13</code> or something.</li>
<li>&ldquo;unlock&rdquo; the pseudoterminal so that we can use it. I have no idea what the point of this is (why is it locked to begin with?) but you have to do it for some reason</li>
<li>open <code>/dev/pts/13</code> (or whatever number we got from <code>ptsname</code>) to get the &ldquo;slave pseudoterminal device&rdquo;</li>
</ul>

<p>What do those <code>ptsname</code> and <code>unlockpt</code> functions do? They just make some
<code>ioctl</code> system calls to the Linux kernel. All of the communication with the
Linux kernel about terminals seems to be through various <code>ioctl</code> system calls.</p>

<p>Here&rsquo;s the code, it&rsquo;s pretty short: (again, I just copied it from <a href="https://github.com/creack/pty/blob/7de28cee0d53510e719c1aeb1850af0fa647c343/pty_linux.go#L41-L54">creack/pty</a>)</p>

<pre><code>func ptsname(f *os.File) string {
	var n uint32
	ioctl(f.Fd(), syscall.TIOCGPTN, uintptr(unsafe.Pointer(&amp;n)))
	return &quot;/dev/pts/&quot; + strconv.Itoa(int(n))
}

func unlockpt(f *os.File) {
	var u int32
	// use TIOCSPTLCK with a pointer to zero to clear the lock
	ioctl(f.Fd(), syscall.TIOCSPTLCK, uintptr(unsafe.Pointer(&amp;u)))
}
</code></pre>

<h3 id="step-2-hook-the-pseudoterminal-up-to-bash">step 2: hook the pseudoterminal up to <code>bash</code></h3>

<p>The next thing we have to do is connect the pseudoterminal to <code>bash</code>. Luckily,
that&rsquo;s really easy &ndash; here&rsquo;s the Go code for it! We just need to start a new
process and set the stdin, stdout, and stderr to <code>tty</code>.</p>

<pre><code>cmd := exec.Command(&quot;bash&quot;)
cmd.Stdin = tty
cmd.Stdout = tty
cmd.Stderr = tty
cmd.SysProcAttr = &amp;syscall.SysProcAttr{
  Setsid: true,
}
cmd.Start()
</code></pre>

<p>Easy! Though &ndash; why do we need this <code>Setsid: true</code> thing, you might ask? Well,
I tried commenting out that code to see what went wrong. It turns out that what
goes wrong is &ndash; Ctrl + C doesn&rsquo;t work anymore!</p>

<p><code>Setsid: true</code> creates a new <strong>session</strong> for the new bash process. But why does
that make <code>Ctrl + C</code> work? How does Linux know which process to send <code>SIGINT</code>
to when you press <code>Ctrl + C</code>, and what does that have to do with sessions?</p>

<h3 id="how-does-linux-know-which-process-to-send-ctrl-c-to">how does Linux know which process to send Ctrl + C to?</h3>

<p>I found this pretty confusing, so I reached for my favourite book for learning
about this kind of thing: <a href="https://man7.org/tlpi/">the linux programming interface</a>, specifically chapter 34 on process groups
and sessions.</p>

<p>That chapter contains a few key facts: (#3, #4, and #5 are direct quotes from the book)</p>

<ol>
<li>Every process has a <strong>session id</strong> and a <strong>process group id</strong> (which may or may not be the same as its PID)</li>
<li>A session is made up of multiple process groups</li>
<li>All of the processes in a session share a single controlling terminal.</li>
<li>A terminal may be the controlling terminal of at most one session.</li>
<li>At any point in time, one of the process groups in a session is the
<strong>foreground process group</strong> for the terminal, and the others are background
process groups.</li>
<li>When you press <code>Ctrl+C</code> in a terminal, SIGINT gets sent to all the processes in the foreground process group</li>
</ol>

<p>What&rsquo;s a process group? Well, my understanding is that:</p>

<ul>
<li>processes in the same pipe <code>x | y | z</code> are in the same process group</li>
<li>processes you start on the same shell line (<code>x &amp;&amp; y &amp;&amp; z</code>) are in the same process group</li>
<li>child processes are by default in the same process group, unless you explicitly decide otherwise</li>
</ul>

<p>I didn&rsquo;t know most of this (I had no idea processes had a session ID!) so this
was kind of a lot to absorb. I tried to draw a sketchy ASCII art diagram of the
situation</p>

<pre><code>(maybe)  terminal --- session --- process group --- process
                               |                 |- process
                               |                 |- process
                               |- process group 
                               |
                               |- process group 
</code></pre>

<p>So when we press Ctrl+C in a terminal, here&rsquo;s what I think happens:</p>

<ul>
<li><code>\x04</code> gets written to the &ldquo;pseudotermimal master&rdquo; of a terminal</li>
<li>Linux finds the <strong>session</strong> for that terminal (if it exists)</li>
<li>Linux find the <strong>foreground process group</strong> for that session</li>
<li>Linux sends <code>SIGINT</code></li>
</ul>

<p>If we don&rsquo;t create a new session for our new bash process, our new pseudoterminal
actually won&rsquo;t have <strong>any</strong> session associated with it, so nothing happens when
we press <code>Ctrl+C</code>. But if we do create a new session, then the new
pseudoterminal will have the new session associated with it.</p>

<h3 id="how-to-get-a-list-of-all-your-sessions">how to get a list of all your sessions</h3>

<p>As a quick aside, if you want to get a list of all the sessions on your Linux
machine, grouped by session, you can run:</p>

<pre><code>$ ps -eo user,pid,pgid,sess,cmd | sort -k3
</code></pre>

<p>This includes the PID, process group ID, and session ID. As an example of the output, here are the two processes in the pipeline:</p>

<pre><code>bork       58080   58080   57922 ps -eo user,pid,pgid,sess,cmd
bork       58081   58080   57922 sort -k3
</code></pre>

<p>You can see that they share the same process group ID and session ID, but of
course they have different PIDs.</p>

<p>That was kind of a lot but that&rsquo;s all we&rsquo;re going to say about sessions and
process groups in this post. Let&rsquo;s keep going!</p>

<h3 id="step-3-set-the-window-size">step 3: set the window size</h3>

<p>We need to tell the terminal how big to be!</p>

<p>Again, I just copied this from <code>creack/pty</code>. I decided to hardcode the size to 80x24.</p>

<pre><code>Setsize(tty, &amp;Winsize{
		Cols: 80,
		Rows: 24,
	})
</code></pre>

<p>Like with getting the terminal&rsquo;s pts filename and unlocking it, setting the
size is just one <code>ioctl</code> system call:</p>

<pre><code>func Setsize(t *os.File, ws *Winsize) {
	ioctl(t.Fd(), syscall.TIOCSWINSZ, uintptr(unsafe.Pointer(ws)))
}
</code></pre>

<p>Pretty simple! We could do something smarter and get the real window size, but
I&rsquo;m too lazy.</p>

<h3 id="step-4-copy-information-between-the-tcp-connection-and-the-pseudoterminal">step 4: copy information between the TCP connection and the pseudoterminal</h3>

<p>As a reminder, our rough steps to set up this remote login server were:</p>

<ol>
<li>create a pseudoterminal for the client to use</li>
<li>start a <code>bash</code> shell process</li>
<li>connect <code>bash</code> to the pseudoterminal</li>
<li>continuously copy information back and forth between the TCP connection and
the pseudoterminal</li>
</ol>

<p>We&rsquo;ve done 1, 2, and 3, now we just need to ferry information between the TCP
connection and the pseudoterminal.</p>

<p>There are two <code>io.Copy</code> calls, one to copy the input <em>from</em> the tcp connection, and one to copy the output <em>to</em> the TCP connection. Here&rsquo;s what the code looks like:</p>

<pre><code>	go func() {
			io.Copy(pty, conn)
	}()
  io.Copy(conn, pty)
</code></pre>

<p>The first one is in a goroutine just so they can both run in parallel.</p>

<p>Pretty simple!</p>

<h3 id="step-5-exit-when-we-re-done">step 5: exit when we&rsquo;re done</h3>

<p>I also added a little bit of code to close the TCP connection when the command exits</p>

<pre><code>go func() {
  cmd.Wait()
  conn.Close()
}()

</code></pre>

<p>And that&rsquo;s it for the server!  You can see all of the Go code here: <a href="https://github.com/jvns/tiny-remote-login/blob/main/server.go">server.go</a>.</p>

<h3 id="next-write-a-client">next: write a client</h3>

<p>Next, we have to write a client. This is a lot easier than the server because we don&rsquo;t need to do quite as much terminal setup. There are just 3 steps:</p>

<ol>
<li>Put the terminal into raw mode</li>
<li>copy stdin/stdout to the TCP connection</li>
<li>reset the terminal</li>
</ol>

<h3 id="client-step-1-put-the-terminal-into-raw-mode">client step 1: put the terminal into &ldquo;raw&rdquo; mode</h3>

<p>We need to put the client terminal into &ldquo;raw&rdquo; mode so that every time you press
a key, it gets sent to the TCP connection immediately. If we don&rsquo;t do this,
everything will only get sent when you press enter.</p>

<p>&ldquo;Raw mode&rdquo; isn&rsquo;t actually a single thing, it&rsquo;s a bunch of flags that you want
to turn off. There&rsquo;s a good tutorial explaining all the flags we have to turn
off called <a href="https://viewsourcecode.org/snaptoken/kilo/02.enteringRawMode.html">Entering raw mode</a>.</p>

<p>Like everything else with terminals, this requires <code>ioctl</code> system calls. In
this case we get the terminal&rsquo;s current settings, modify them, and save the old
settings so that we can restore them later.</p>

<p>I figured out how to do this in Go by going to <a href="https://grep.app">https://grep.app</a> and typing in
<code>syscall.TCSETS</code> to find some other Go code that was doing the same thing.</p>

<pre><code>func MakeRaw(fd uintptr) syscall.Termios {
	// from https://github.com/getlantern/lantern/blob/devel/archive/src/golang.org/x/crypto/ssh/terminal/util.go
	var oldState syscall.Termios
	ioctl(fd, syscall.TCGETS, uintptr(unsafe.Pointer(&amp;oldState)))

	newState := oldState
	newState.Iflag &amp;^= syscall.ISTRIP | syscall.INLCR | syscall.ICRNL | syscall.IGNCR | syscall.IXON | syscall.IXOFF
	newState.Lflag &amp;^= syscall.ECHO | syscall.ICANON | syscall.ISIG
	ioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&amp;newState)))
	return oldState
}
</code></pre>

<h3 id="client-step-2-copy-stdin-stdout-to-the-tcp-connection">client step 2: copy stdin/stdout to the TCP connection</h3>

<p>This is exactly like what we did with the server. It&rsquo;s very little code:</p>

<pre><code>go func() {
		io.Copy(conn, os.Stdin)
	}()
	io.Copy(os.Stdout, conn)
</code></pre>

<h3 id="client-step-3-restore-the-terminal-s-state">client step 3: restore the terminal&rsquo;s state</h3>

<p>We can put the terminal back into the mode it started in like this (another <code>ioctl</code>!):</p>

<pre><code>func Restore(fd uintptr, oldState syscall.Termios) {
	ioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&amp;oldState)))
}
</code></pre>

<h3 id="we-did-it">we did it!</h3>

<p>We have written a tiny remote login server that lets anyone log in! Hooray!</p>

<p>Obviously this has zero security so I&rsquo;m not going to talk about that aspect.</p>

<h3 id="it-s-running-on-the-public-internet-you-can-try-it-out">it&rsquo;s running on the public internet! you can try it out!</h3>

<p>For the next week or so I&rsquo;m going to run a demo of this on the internet at
<code>tetris.jvns.ca</code>. It runs tetris instead of a shell because I wanted to avoid
abuse, but if you want to try it with a shell you can run it on your own
computer :).</p>

<p>If you want to try it out, you can use <code>netcat</code> as a client instead of the
custom Go client program we wrote, because copying information to/from a TCP
connection is what netcat does. Here&rsquo;s how:</p>

<pre><code>stty raw -echo &amp;&amp; nc tetris.jvns.ca 7777 &amp;&amp; stty sane
</code></pre>

<p>This will let you play a terminal tetris game called <code>tint</code>.</p>

<p>You can also use the <a href="https://github.com/jvns/tiny-remote-login/blob/main/client.go">client.go program</a> and run <code>go run client.go tetris.jvns.ca 7777</code>.</p>

<h3 id="this-is-not-a-good-protocol">this is not a good protocol</h3>

<p>This protocol where we just copy bytes from the TCP connection to the terminal
and nothing else is not good because it doesn&rsquo;t allow us to send over
information information like the terminal or the actual window size of the
terminal.</p>

<p>I thought about implementing telnet&rsquo;s protocol so that we could use telnet as a
client, but I didn&rsquo;t feel like figuring out how telnet works so I didn&rsquo;t. (the
server 30% works with telnet as is, but a lot of things are broken, I don&rsquo;t
quite know why, and I didn&rsquo;t feel like figuring it out)</p>

<h3 id="it-ll-mess-up-your-terminal-a-bit">it&rsquo;ll mess up your terminal a bit</h3>

<p>As a warning: using this server to play tetris will probably mess up your
terminal a bit because it sets the window size to 80x24. To fix that I just
closed the terminal tab after running that command.</p>

<p>If we wanted to fix this for real, we&rsquo;d need to restore the window size after
we&rsquo;re done, but then we&rsquo;d need a slightly more real protocol  than &ldquo;just
blindly copy bytes back and forth with TCP&rdquo; and I didn&rsquo;t feel like doing that.</p>

<p>Also it sometimes takes a second to disconnect after the program exits for some
reason, I&rsquo;m not sure why that is.</p>

<h3 id="other-tiny-projects">other tiny projects</h3>

<p>That&rsquo;s all! There are a couple of other similar toy implementations of programs
I&rsquo;ve written here:</p>

<ul>
<li><a href="https://jvns.ca/blog/2022/03/23/a-toy-version-of-tls/">toy tls 1.3 implementation</a></li>
<li><a href="https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/">toy dns resolver</a></li>
</ul>


guid: a20ef99a1c72873dd88fd038b49aca4d80371b0c
title: No, you cannot trust third party code without reading it first
url: https://unixsheikh.com/articles/no-you-cannot-trust-third-party-code-without-reading-it-first.html
feedurl: https://unixsheikh.com/feed.rss
content:
For more than a decade I have been thundering against a lot of the bad practices that have permeated the software development industry, one such practice is to blindly trust code when using third party libraries, frameworks or packages. For about the same amount of time I have listened to all the reasons why time is money and we need to build something quickly, and we haven't got the time to do security or X, Y and Z. But alas, now such companies are beginning to pay the price, a very costly and extremely damaging price!


guid: f5f16f410ee505bbc7ad0a17fada9b1a67465241
title: "Zero trust" is being hijacked by the big corporate hype machine
url: https://unixsheikh.com/articles/zero-trust-is-being-hijacked-by-the-big-corporate-hype-machine.html
feedurl: https://unixsheikh.com/feed.rss
content:
The term zero trust is a security model, also known as "perimeterless security" that has been known for a long time and that was e.g. implemented internally at Google in 2009. In recent years, mainly due to the high amount of security problems the industry is facing, the term gained renewed focus and popularity when cybersecurity researchers at National Institute of Standards and Technology (NIST) released their Zero Trust Architecture (PDF) publication. 


guid: https://www.joelonsoftware.com/?p=3246
title: The Stack Overflow Age
author: Joel Spolsky
url: https://www.joelonsoftware.com/2018/04/06/the-stack-overflow-age/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>Hi, everyone! A lot of stuff has happened since I was writing all those blog posts about Aeron chairs 18 years ago. Some of those blog posts are old enough to go to college.</p>
<p>And, also: <a href="https://stackoverflow.com/">Stack Overflow</a> will be ten years old soon! Wow! So I thought it would be cool to get the old band back together for a little reunion tour over the next few weeks. I want to catch you all up on some stuff but mostly I want to tell the story of Stack Overflow in a not-completely-disorganized way. With some perspective, it’s clearer now what we did right and what we messed up, so I’ll try to cover the good and the bad over a series of blog posts.</p>
<p>And, also: we’re just a few weeks away from launching Stack Overflow Teams, the biggest upgrade to Stack Overflow ever, so that’s going to be really cool. I’ll get to that in a future blog post!</p>
<p>Today is chapter one. I want to talk a little bit about what it was like for developers before Stack Overflow, the problem that Stack Overflow tried to solve, and early origins.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?ssl=1"><img loading="lazy" class="alignright wp-image-3253 size-medium" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?resize=225%2C300&#038;ssl=1" alt="" width="225" height="300" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?resize=225%2C300&amp;ssl=1 225w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?resize=768%2C1024&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?resize=1200%2C1600&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/IMG_5567.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 225px) 100vw, 225px" data-recalc-dims="1" /></a>In the early days of the Internet, before the Web, there was a system called <a href="https://today.duke.edu/2010/05/usenet.html">Usenet</a> which created primitive online discussion forums. When programmers had problems with their code, they could ask a question on a Usenet forum. (They were technically called newsgroups, not forums (even though they had nothing to do with news. (You couldn’t even get news on Usenet.)))</p>
<p>As soon as the world wide web became a thing, Usenet was immediately technically obsolete. We programmers started asking about our problems on various web-based forums, of which there were thousands.</p>
<p>One of the biggest such forums was called Experts Exchange. The first version of Experts Exchange was not successful financially. Apparently they went bankrupt in 2001. Eventually new owners bought the assets and resurrected the site with a clever business model: charging money to read answers.</p>
<p>This actually fixed the business, which started making money, but it caused some problems.</p>
<p>The first problem was that programmers with problems would search on Google, not on Experts Exchange. And Google only knows about free, open websites, not websites that you have to pay to access. So EE did a bamboozle: when the Google Robot came by, they showed it the full question and its answers. But when regular people went to the same page, they saw the answers were scrambled, with instructions to pay (I think it was about $250 a year) to see the results. Most programmers couldn’t be bothered.</p>
<p>The second problem was that EE let you get a free membership if you answered a certain number of questions. As it turned out, the people who were most desperate for free memberships were not exactly the best programmers in the world, and they wrote low quality answers to questions just to get those free memberships. And the quality of answers on the site went down.</p>
<p>For a long time (at least five years, I think) programmers would constantly come across EE in the Google search results, try to click on them, discover that it was a pay site, grumble, and just go back to Google and try to find an answer for free.</p>
<p>And I kept thinking, how hard is it to run a discussion forum on the Internet? For fudge sake, I had written one in Visual Basic in a weekend. (Not kidding, actually. Yeah I know that I am always saying “I could do that in a weekend in Visual Basic” when developers tell me some feature is going to take a year. This is why). So I was confident that it was only a matter of time before one of the 9,000,000 smart programmers in the world decided to route around this EE damage and make a free forum.</p>
<p>You know what? Nobody ever did. I kept waiting.</p>
<p>Another thing I wrote in a weekend (well, to be precise: a fortnight (shut up, I’m telling this lie)) was a <a href="https://www.joelonsoftware.com/2006/09/05/introducing-jobsjoelonsoftwarecom/">job listing board for this blog</a>. And in the first month of running that job board I think we sold about $90,000 of job listings. Huzzah! And then I thought, wow, if we smashed these ideas together—replace Experts Exchange with a free site, and pay for it with job listings—we could undo the damage to the internet and let developers get work done again.</p>
<p>I kept thinking “Man, this is so obvious, somebody is going to do it.”</p>
<p>And they never did.</p>
<p>And I went to one of the programmers at Fog Creek, and explained my idea, and he was like “yeah yeah sounds like a great idea, but I really like working on FogBugz.”</p>
<p>And more time went by.</p>
<p>And eventually, early in 2008, a developer/blogger named <a href="https://blog.codinghorror.com/">Jeff Atwood</a> called me up, and said, “Hey Joel, I’m thinking of quitting my day job to be a Pro Blogger; you’re a blogger: what do you think?”</p>
<p>And I said, “Jeff, I’ve got a better idea” and I told him about the idea to combine the job listings with the Q&amp;A site for developers, and, it took more than a weekend, but eventually I convinced him. We started talking about all the ways our Q&amp;A site would be amazing. Jeff started working on the code in April 2008, recruited two other programmers to join him (Geoff and Jarrod, who are still here), and the three of them heroically launched what became Stack Overflow in September 2008.</p>
<p>&nbsp;</p>
<p><img loading="lazy" class="aligncenter" src="https://i0.wp.com/i.stack.imgur.com/GjnH9.png?resize=427%2C446&#038;ssl=1" alt="The original Stack Overflow" width="427" height="446" data-recalc-dims="1" /></p>
<p>&nbsp;</p>
<p>And thus began the Stack Overflow Age.</p>
<p>Stack Overflow was better because it was free, but it had a ton of other “innovations” (which I put in quotes because we stole them from other Internet pioneers) which made it a much, much better site for getting answers to programming questions.</p>
<p>We wanted the whole thing to be a fun game, with incentives to answer questions, so we had a reputation system. The more you answer, the more reputation you earn. The reputation idea had been seen before on sites like Slashdot and Reddit.</p>
<p>As you earn reputation, you also earn moderation privileges on the site. So the site actually moderates itself, which is pretty cool.</p>
<p>Instead of putting all the Java programmers in one little forum and all the C++ programmers in another, we dumped everyone together and just let them tag their questions. This idea was stolen from <a href="https://www.flickr.com/">flickr</a> (remember flickr?) who, I think, stole it from del.icio.us (now gone)—who knows, anyway, the point is, tags were the new hotness and made Stack Overflow work great.</p>
<p>Most importantly, we realized that each question is asked by one person but the answers are seen by thousands of people who found it through a search. So we decided to optimize everything to be useful for the thousands, not the individual. We literally have 1000 visitors for every person who asks a question. That’s why we sort the answers by votes. It’s also why we optimize for questions and answers that will be helpful to other people, later.</p>
<p>Interestingly, when Jeff and I started Stack Overflow, we didn’t really care if it was a business and we didn’t need it to be a big profitable success. We created it because the internet sucked for programmers and we needed to make it better. We thought the job listings would pay the bills, and we’d fix the internet, and that was all we cared about and it’s what motivated us to work so hard.</p>
<p>Of course, it turned out a lot bigger than we thought it would. The company today has 250 employees, is profitable, and has made it possible for millions of people to learn how to code and to deal with the new, super-complicated world of APIs and frameworks that we live in. But we just wanted to fix the internet.</p>
<p>I have met a lot of people who started businesses because they wanted to start a business. <a href="http://startupclass.samaltman.com/courses/lec03/">Paul Graham calls this “Playing House.”</a> And they didn’t really care what the business did; they just wanted to “be entrepreneurs.” Which is weird, because being an entrepreneur really sucks. It’s really hard to get through all the extraordinary difficulty, pain, and stress of starting a company if you’re not super, super motivated to solve a problem for the world.</p>
<p>The entrepreneurs who succeed do so because it is incredibly important to them a thing exist in the world, and it does not exist, so they work like crazy until it does. When we started Stack Overflow we didn’t expect it to be a big business; we just wanted there to be someplace where developers could get help to daily problems, while showing off how smart they were helping other developers.</p>
<p>Ok, that’s chapter one. I&#8217;ve got a lot more to talk about. In the next installment, I’ll talk more about <a href="https://www.joelonsoftware.com/2018/04/13/gamification/">how Stack Overflow’s light dusting of gamification made it really take off.</a></p>


guid: https://www.joelonsoftware.com/?p=3260
title: A Dusting of Gamification
author: Joel Spolsky
url: https://www.joelonsoftware.com/2018/04/13/gamification/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p><em>[This is the second in a series of posts about Stack Overflow. The first one is <a href="https://www.joelonsoftware.com/2018/04/06/the-stack-overflow-age/">The Stack Overflow Age</a>.]</em></p>
<p>Around 2010 the success of Stack Overflow had led us into some conversations with VCs, who wanted to invest.</p>
<p><img loading="lazy" class="aligncenter wp-image-3273" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?resize=500%2C667&#038;ssl=1" alt="at the Getty" width="500" height="667" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?resize=768%2C1024&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?resize=225%2C300&amp;ssl=1 225w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?resize=1200%2C1600&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-09-12.54.00.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 500px) 100vw, 500px" data-recalc-dims="1" /></p>
<p>The firm that eventually invested in us, <a href="https://www.usv.com/">Union Square Ventures</a>, told us that they were so excited by the power of gamification that they were only investing in companies that incorporated some kind of game play.</p>
<p>For example, Foursquare. Remember Foursquare? It was all about making your normal post-NYU life of going to ramen noodle places and dive bars into a fun game that incidentally generated wads of data for marketers. Or Duolingo, which is a fun app with flash cards that teaches you foreign languages. Those were other USV investments from that time period.</p>
<p>At the time, I had to think for a minute to realize that Stack Overflow has “gamification&#8221; too. Not a ton. Maybe a <em>dusting</em> of gamification, most of it around reputation.</p>
<p>Stack Overflow reputation started as a very simple score. The original idea was just that you would get 10 points when your answers were upvoted. Upvotes do two things. They get the most useful answers to the top, signaling that other developers who saw this answer thought it was good. They also send the person who wrote the answer a real signal that their efforts helped someone. This can be incredibly motivating.</p>
<p>You would lose points if your questions were downvoted, but you actually only lose 2 points. We didn&#8217;t want to punish you so much as we wanted to show other people that your answer was wrong. And to avoid abuse, we actually make you <em>pay</em> one reputation point to downvote somebody, so you better really mean it. That was pretty much the whole system.</p>
<p>Now, this wasn&#8217;t an original idea. It was originally inspired by Reddit Karma, which started out as an integer that appeared in parentheses after your handle. If you posted something that got upvoted, your karma went up as a &#8220;reward.&#8221; That was it. Karma <a href="https://www.zdnet.com/article/collaborative-filtering-comparing-reddits-karma-system-to-digg/">didn’t do a single thing</a> but still served as a system for reward and punishment.</p>
<p>What reputation and karma do is send a message that this is a community with norms, it&#8217;s not just a place to type words onto the internet. (That would be 4chan.) We don&#8217;t really exist for the purpose of letting you exercise your freedom of speech. You can get your freedom of speech somewhere else. Our goal is to get the best answers to questions. All the voting makes it clear that we have standards, that some posts are better than others, and that the community itself has some norms about what&#8217;s good and bad that they express through the vote.</p>
<p>It&#8217;s not a perfect system (more on the problems in a minute), but it&#8217;s a reasonable first approximation.</p>
<p>By the way, Alexis Ohanian and Steve Huffman, the creators of Reddit, were themselves inspired by a more primitive karma system, on Slashdot. This system had real-world implications. You didn’t get karma so that other people could see your karma; you got karma so that the system knew you weren’t a spammer. If a lot of your posts had been flagged for abuse, your karma would go down and you might lose posting or moderation privileges. But you weren’t really supposed to show off your high karma. “Don&#8217;t worry too much about it; it&#8217;s just an integer in a database,” Slashdot told us.</p>
<p>To be honest, it was initially surprising to me that you could just print a number after people’s handles and they would feel rewarded. Look at me! Look at my four digit number! But it does drive a tremendous amount of good behavior. Even people who aren’t participating in the system (by working to earn reputation) buy into it (e.g., by respecting high-reputation users for their demonstrated knowledge and helpfulness).</p>
<p>But there&#8217;s still something of a mystery here, which is why earning &#8220;magic internet points&#8221; is appealing to anyone.</p>
<p>I think the answer is that it&#8217;s nice to know that you&#8217;ve made a difference. You toil away in the hot kitchen all day and when you serve dinner it&#8217;s nice to hear a compliment or two. If somebody compliments you on the extra effort you put into making radish roses, you&#8217;re going to be very happy.</p>
<p>This is a part of a greater human need: to make an impact on the world, and to know that you&#8217;re contributing and being appreciated for it. Stack Overflow&#8217;s reputation system serves to recognize that you&#8217;re a human being and we are super thankful for your contribution.</p>
<p><img loading="lazy" class="aligncenter wp-image-3276 size-large" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?resize=730%2C548&#038;ssl=1" alt="in Utah" width="730" height="548" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2018-01-11-13.46.38.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></p>
<p>That said, there is a dark side to gamification. It&#8217;s not 100% awesome.</p>
<p>The first problem we noticed is that it&#8217;s very nice to get an upvote, but getting a downvote feels like a slap in the face. Especially if you don&#8217;t understand why you got the downvote, or if you don&#8217;t agree. Stack Overflow&#8217;s voting has made many people unhappy over the years, and there are probably loads of people who felt unwelcome and who don&#8217;t participate in Stack Overflow as a result. (Here&#8217;s an old blog post explaining <a href="https://stackoverflow.blog/2009/03/09/the-value-of-downvoting-or-how-hacker-news-gets-it-wrong/">why we didn&#8217;t just eliminate downvotes</a>).</p>
<p>There&#8217;s another problem, which is that, to the extent that the gamification in Stack Overflow makes the site feel less inclusive and welcoming to many people, it is disproportionately off-putting to programmers from underrepresented groups. While Stack Overflow does have many amazing, high reputation contributors who are women or minorities, we&#8217;ve also heard from too many who were apprehensive about participating.</p>
<p>These are big problems. There&#8217;s a lot more we can and will say about that over the next few months, and we&#8217;ve got a lot of work ahead of us trying to make Stack Overflow a more inclusive and diverse place so we can improve the important service that it provides to developers everywhere.</p>
<p>Gamification can shape behavior. It can guide you to do certain things in certain ways, and it can encourage certain behaviors. But it&#8217;s a very weak force. You can’t do that much with gamification. You certainly can’t get people to do something that they’re not interested in doing, anyway. I’ve heard a lot of crazy business plans that are pinning rather too high hopes on gamification as a way of getting people to go along with some crazy scheme that the people won’t want to go along with. Nobody’s going to learn French just to get the Duolingo points. But if you are learning French, and you are using Duolingo, you might make an effort to open the app every day just to keep your streak going.</p>
<p>I&#8217;ve got more posts coming! The next one will be about the obsessive way Stack Overflow was designed for the artifact, in other words, we optimized everything to create amazing results for developers with problems arriving from Google, not just to answer questions that people typed into our site.</p>
<p>&nbsp;</p>


guid: https://www.joelonsoftware.com/?p=3282
title: Strange and maddening rules
author: Joel Spolsky
url: https://www.joelonsoftware.com/2018/04/23/strange-and-maddening-rules/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>There&#8217;s this popular idea among developers that when you face a problem with code, you should get out a rubber duck and explain, to the duck, exactly how your code was supposed to work, line by line, what you expected to see, what you saw instead, etc. Developers who try this report that the very act of explaining the problem in detail to an inanimate object often helps them find the solution.</p>
<p><a href="https://meta.stackexchange.com/questions/308564/stack-exchange-has-been-taken-over-by-a-rubber-duck"><img loading="lazy" class="size-large aligncenter" src="https://upload.wikimedia.org/wikipedia/en/e/ef/StackExchange_Rubber_Duck_Avatar_April_Fools_2018.png" alt="Stack Overflow April Fools Joke 2018" width="270" height="270" /></a>This is one of many tricks to solving programming problems on your own. Another trick is <a href="http://markheath.net/post/effective-debugging-with-divide-and-conquer">divide and conquer debugging</a>. You can&#8217;t study a thousand lines of code to find the one bug. But you can divide them in half and quickly figure out if the problem happens in the first half or the second half. Keep doing this five or six times and you&#8217;ll pinpoint the single line of code with the problem.</p>
<p>It&#8217;s interesting, with this in mind, to read <a href="https://stackoverflow.com/users/22656/jon-skeet?tab=profile">Jon Skeet&#8217;s</a> checklist for <a href="https://codeblog.jonskeet.uk/2012/11/24/stack-overflow-question-checklist/">writing the perfect question</a>. One of the questions Jon asks is &#8220;Have you read the whole question to yourself carefully, to make sure it makes sense and contains enough information for someone coming to it without any of the context that you already know?&#8221; That is essentially the Rubber Duck Test. Another question is &#8220;If your question includes code, have you written it as a short but complete program?&#8221; Emphasis on the <em>short</em>—that is essentially a test of whether or not you tried divide and conquer.</p>
<p>What Jon&#8217;s checklist can do, in the best of worlds, is to help people try the things that experienced programmers may have already tried, before they ask for help.</p>
<p>Sadly, not everybody finds his checklist. Maybe they found it and they don&#8217;t care. They&#8217;re having an urgent problem with code; they heard that Stack Overflow could help them; and they don&#8217;t have time to read some nerd&#8217;s complicated protocol for requesting help.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?ssl=1"><img loading="lazy" class="aligncenter size-large wp-image-3352" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?resize=730%2C411&#038;ssl=1" alt="" width="730" height="411" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?resize=1024%2C576&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?resize=300%2C169&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?resize=768%2C432&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?resize=1200%2C675&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?w=1920&amp;ssl=1 1920w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/NYC.png?w=1460&amp;ssl=1 1460w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p>One of the frequent debates about Stack Overflow is whether the site needs to be open to questions from programming novices.</p>
<p>When Jeff and I were talking about the initial design of Stack Overflow, I told him about this popular Usenet group for the C programming language in the 1980s. It was called <a href="https://groups.google.com/forum/#!forum/comp.lang.c">comp.lang.c.</a></p>
<p>C is a simple and limited programming language. <a href="http://repo.or.cz/w/tinycc.git">You can get a C compiler that fits in 100K.</a> So, when you make a discussion group about C, you quickly run out of things to talk about.</p>
<p>Also. In the 1990s, C was a common language for undergraduates who were learning programming. And, in fact, said undergraduates would have <a href="https://groups.google.com/forum/#!searchin/comp.lang.c/spolsky%7Csort:date/comp.lang.c/BKhAacGNtZI/GLSW-QG_O3MJ">very basic problems</a> in C. And they might show up on comp.lang.c asking their questions.</p>
<p>And the old-timers on comp.lang.c were bored. <i>So</i> bored. Bored of the undergraduates showing up every September wondering why they can’t return a local char array from a function et cetera, et cetera, ad nauseum. Every damn September.</p>
<p>The old timers invented the concept of FAQs. They used them to say “please don’t ask things that have been asked before, ever, in the history of Usenet” which honestly meant that the only questions they really wanted to see were so bizarre and so esoteric that they were really enormously boring to 99% of working C programmers. The newsgroup languished because it catered only to the few people that had been there for a decade.</p>
<p>Jeff and I talked about this. What did we think of newbie questions?</p>
<p>We decided that newbies had to be welcome. Nothing was too &#8220;beginner&#8221; to be a reasonable question on Stack Overflow&#8230; as long as you did some homework before asking the question.</p>
<p>We understood that this might mean that some of the more advanced people might grow bored with duplicate, simple questions, and move on. We thought that was fine: Stack Overflow doesn’t have to be a lifetime commitment. You’re welcome to get bored and move on if you think that the newbies keep asking why they can’t return local char arrays (“but it works for me!”) and you would rather devote the remaining short years of your life to something more productive, like sorting your record albums.</p>
<p>The mere fact that you are a newbie doesn’t mean that your question doesn&#8217;t belong on Stack Overflow. To prove the point, I asked &#8220;<a href="https://stackoverflow.com/questions/1003841/how-do-i-move-the-turtle-in-logo">How do you move the turtle in Logo</a>,&#8221; hoping to leave behind evidence that the site designers wanted to allow absolute beginners.</p>
<p>Thanks to the law of unintended consequences, this caused a lot of brouhaha, but not because the question was too easy. The real problem there was that I was asking the question in bad faith. Jeff Atwood <a href="https://meta.stackexchange.com/questions/158289/why-is-the-how-to-move-the-turtle-in-logo-question-closed#comment457933_158334">explained</a> it: “Simple is fine. No effort and research is not.” (<a href="https://en.wikipedia.org/wiki/Wikipedia:Do_not_disrupt_Wikipedia_to_illustrate_a_point">Also this</a>.)</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?ssl=1"><img loading="lazy" class="aligncenter size-large wp-image-3355" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?resize=730%2C411&#038;ssl=1" alt="" width="730" height="411" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?resize=1024%2C576&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?resize=300%2C169&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?resize=768%2C432&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?resize=1200%2C675&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?w=1920&amp;ssl=1 1920w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/StBarts.png?w=1460&amp;ssl=1 1460w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p><span style="font-weight: 400;">To novices, the long bureaucratic rigmarole associated with asking your first question on Stack Overflow can feel either completely unnecessary, or just plain weird. It’s like Burning Man. You just want to go to a nice glittery dance party in the desert, but the Burning People are yammering on about their goddamn </span><a href="https://burningman.org/culture/philosophical-center/10-principles/"><span style="font-weight: 400;">10 principles</span></a><span style="font-weight: 400;">, and “radical self-expression” and so on and so forth, and </span><i><span style="font-weight: 400;">therefore</span></i><span style="font-weight: 400;"> after washing your dishes you must carefully save the dirty dishwater like a cherished relic and remove every drop of it from the Playa, bringing it home with you, in your check-in luggage if necessary. Every community has lots of rules and when you join the community they either seem strange and delightful or, if you’re just desperately trying to get some code to work, they are strange and maddening.</span></p>
<p><span style="font-weight: 400;">A lot of the rules that are important to make Burning Man successful are seemingly arbitrary, but they’re still necessary. The US Bureau of Land Management which makes the desert available for Burning Man requires that no contaminated water be poured out on the ground because the clay dirt doesn’t really absorb it so well and it can introduce all kinds of disease and whatnot, but who cares because Burning Man simply will not be allowed to continue if the participants don’t pack out their used water.</span></p>
<p><span style="font-weight: 400;">Similarly for Stack Overflow. We don’t allow, say, questions that are too broad (“How do I make a program?”). Our general rule is that if the correct length of an answer is </span><i><span style="font-weight: 400;">a whole book </span></i><span style="font-weight: 400;">you are asking too much. These questions feel like showing up on a medical website and saying something like &#8220;I think my kidney has been hurting. How can I remove it?&#8221; It&#8217;s crazy—and incidentally, insulting to the people who spent ten years in training learning to be surgeons.</span></p>
<p><span style="font-weight: 400;">One thing I’m very concerned about, as we try to educate the next generation of developers, and, importantly, get more diversity and inclusiveness in that new generation, is what obstacles we’re putting up for people as they try to learn programming. In many ways Stack Overflow’s specific rules for what is permitted and what is not are obstacles, but an even bigger problem is rudeness, snark, or condescension that newcomers often see.</span></p>
<p>I care a lot about this. Being a developer gives you an unparalleled opportunity to <a href="https://www.joelonsoftware.com/2016/12/09/developers-are-writing-the-script-for-the-future/">write the script for the future</a>. All the flak that Stack Overflow throws in the face of newbies trying to become developers is actively harmful to people, to society, and to Stack Overflow itself, by driving away potential future contributors. And programming is hard enough; we should see our mission as making it easier.</p>
<p>We&#8217;re planning a lot of work in this area for the next year. We can&#8217;t change everybody and we can&#8217;t force people to be nice. But I think we can improve some aspects of the Stack Overflow user interface to encourage better behavior, for example, we could improve the prompts we provide on the &#8220;Ask Question&#8221; page, and we could provide more tools for community moderation of comments where the snark currently runs unchecked.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?ssl=1"><img loading="lazy" class="aligncenter size-large wp-image-3364" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?resize=730%2C548&#038;ssl=1" alt="" width="730" height="548" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/2016-09-10-21.38.14-1.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p>We&#8217;re also working on new features that will let you direct your questions to a private, smaller group of people on your own team, which may bring some of the friendly neighborhood feel to the big city of Stack Overflow.</p>
<p><span style="font-weight: 400;">Even as we try to make Stack Overflow more friendly, our primary consideration at Stack Overflow has been to build the world’s greatest resource for software developers. The average programmer, in the world, has been helped by Stack Overflow 340 times. That’s the real end-game here. There are other resources for learning to program and getting help, but there’s only one site in the world that developers trust this much, and that is worth preserving—the programming equivalent to the Library of Congress.</span></p>


guid: https://www.joelonsoftware.com/?p=3411
title: Announcing Stack Overflow for Teams
author: Joel Spolsky
url: https://www.joelonsoftware.com/2018/05/03/announcing-stack-overflow-for-teams/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>Hey, we have a new thing for you today!</p>
<p>Today’s new thing is called <a href="https://stackoverflow.com/teams">Stack Overflow for Teams</a>. It lets you set up a private place on Stack Overflow where you can ask questions that will only be visible to members of your team, company, or organization. It is a paid service, but it’s not expensive.</p>
<p>I meet people who use Stack Overflow every single day, but a lot of them tell me they have never needed to post their own question. “All the questions are already answered!” they say. Mission accomplished, I guess!</p>
<p>Still, when I think about what questions developers have every day, only the ones that have to do with public stuff can be asked on Stack Overflow. Maybe you don&#8217;t have a question about Python or Android&#8230; maybe you want to ask something about your team&#8217;s own code base!</p>
<p>That’s the idea behind Stack Overflow Teams.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?ssl=1"><img loading="lazy" class="aligncenter wp-image-3424 size-large" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?resize=730%2C255&#038;ssl=1" alt="Helicopters" width="730" height="255" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?resize=1024%2C357&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?resize=300%2C104&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?resize=768%2C267&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?resize=1200%2C418&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/05/Helicopters.png?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p>Quick background: every development team since the beginning of time has been trying to figure out how to get institutional knowledge out of people&#8217;s heads and into written, searchable form where everyone can find it. Like new members of the team. And old members of the team working on new parts of the code. And people who forgot what they did three years ago and now have questions about <em>their own code</em>.</p>
<p>For a while developers thought wikis might be the solution. Anyone who has used a wiki for this purpose has probably discovered that not very much knowledge actually makes it into the wiki, and what does is not particularly useful, doesn&#8217;t get updated, and honestly it just feels like a bunch of homework to write a bunch of wiki documentation about your code when you don&#8217;t know if it will ever help anyone.</p>
<p>Another solution being sold today is the idea of having some kind of online IRC-style chat rooms, and hoping that by searching those chat archives, you can find &#8220;institutional knowledge.&#8221; Ha ha ha! Even if that works, all you really find is the history of some conversation people had. It might have clues but it&#8217;s not knowledge.</p>
<p>But you know what does work? A Q&amp;A system. Like Stack Overflow.</p>
<p>Why? Because unlike wikis, you don&#8217;t write documentation in the hopes that one day it might help someone. You answer questions that are going to help someone <em>immediately.</em> And you can stop answering the minute you get the green checkmark that shows that you solved their problem.</p>
<p>And unlike chatrooms, searching actually works. It finds you a question and its answers, not a conversation-captured-in-amber.</p>
<p>This is why Stack Overflow worked so much better on the public internet than the previous generation of discussion forums, and we think that it will work for all the same reasons with teams&#8217; <em>proprietary</em> questions and answers.</p>
<p>When you join a team, you&#8217;ll see your team&#8217;s private questions right on <a href="https://stackoverflow.com/">stackoverflow.com</a> (although they actually live in a separate database for security). Your teams are listed in the left hand navbar.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams1.png?ssl=1"><img loading="lazy" class="aligncenter wp-image-3417 size-full" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams1.png?resize=730%2C584&#038;ssl=1" alt="Screen Shot" width="730" height="584" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams1.png?w=943&amp;ssl=1 943w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams1.png?resize=300%2C240&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams1.png?resize=768%2C614&amp;ssl=1 768w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p>Everything else works pretty much &#8230; like you would expect. When you ask a question, you can direct it to your team or to the whole world. The UI makes it very clear whether you are posting publicly or privately. If you are asking a question of your team, there&#8217;s a <em>Notify</em> field so you can type the names of some people who might be able to answer the question, and they&#8217;ll hear about it right away.</p>
<p><a href="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams2.png?ssl=1"><img loading="lazy" class="aligncenter wp-image-3418 size-full" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams2.png?resize=730%2C744&#038;ssl=1" alt="Screen shot" width="730" height="744" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams2.png?w=859&amp;ssl=1 859w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams2.png?resize=294%2C300&amp;ssl=1 294w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2018/04/Teams2.png?resize=768%2C783&amp;ssl=1 768w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></a></p>
<p>When you search, you can search everywhere, or just within your team. You can set up tags that are specific to your team, too.</p>
<p>The pricing is designed to be “no-brainer” pricing, starting at just $10 per month for the first ten users.</p>
<p>I think <a href="https://stackoverflow.com/teams">Stack Overflow for Teams</a> is going to be almost as important to developers&#8217; daily work as public Stack Overflow. It brings Stack Overflow&#8217;s uniquely powerful system to every developer question, not just the things that can be discussed in public. You can stop asking your teammates questions in email (where they help nobody else) or in chatrooms (where they are impossible to find) and start building your own private knowledge base to document your code and answer future teammates&#8217; questions before they have them.</p>


guid: https://www.joelonsoftware.com/?p=3535
title: The next CEO of Stack Overflow
author: Joel Spolsky
url: https://www.joelonsoftware.com/2019/03/28/the-next-ceo-of-stack-overflow/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>Big news! We’re looking for a new CEO for Stack Overflow. I’m stepping out of the day-to-day and up to the role of Chairman of the Board.</p>



<p>Stack Overflow has been around for more than a decade. As I look back, it’s really amazing how far it has come. &nbsp;</p>



<p>Only six months after we had launched Stack Overflow, my co-founder Jeff Atwood and I were invited to speak at a Microsoft conference for developers in Las Vegas. We were there, I think, to demonstrate that you could use their latest ASP.NET MVC technology on a real website without too much of a disaster. (In fact .NET has been a huge, unmitigated success for us, but you kids go ahead and have fun with whatever platform you want mkay? They’re all great, or, at least, above-average).</p>



<p>It was a giant conference, held at the Venetian Hotel. This hotel was so big that other <em>hotels</em> stay there when they go on vacation. The main ballroom was the size of, approximately, Ireland. I later learned there were 5,000 developers in that room.</p>



<p>I thought it would be a fun thing to ask the developers in the room how many of them had visited Stack Overflow. As I remember, Jeff was very much against this idea. “Joel,” he said, “That is going to be embarrassing and humiliating. Nobody is going to raise their hand.”</p>



<p>Well, I asked it anyway. And we were both surprised to see about one-third of the hands go up. We were really making an impact! That felt really good.</p>



<p>Anyway, I tried that trick again whenever I spoke to a large audience. It doesn’t work anymore. Today, audiences just laugh. It’s like asking, “Does anyone use gravity? Raise your hand if you use gravity.”</p>



<figure class="wp-block-image"><img loading="lazy" width="4032" height="3024" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?fit=730%2C548&amp;ssl=1" alt="" class="wp-image-3538" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?w=4032&amp;ssl=1 4032w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_9111.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" /></figure>



<p>Where are we at after 11 years? Practically every developer in the world uses Stack Overflow. Including the Stack Exchange network of 174 sites, we have over 100 million monthly visitors. Every month, over 125,000 wonderful people write answers. According to Alexa, <a href="https://www.alexa.com/siteinfo/stackoverflow.com">stackoverflow.com is one of the top 50 websites in the world</a>. (That’s without even counting the Stack Exchange network, which is almost as big.) And every time I see a developer write code, they’ve got Stack Overflow open in one of their browser windows. Oh and—hey!—we do not make you sign up or pay to see the answers.</p>



<p>The company has been growing, too. Today we are profitable. We have almost 300 amazing employees worldwide and booked $70m in revenue last year. We have talent, advertising, and software products. The SaaS products (Stack Overflow for <a href="https://stackoverflow.com/teams">Teams</a> and <a href="https://stackoverflow.com/enterprise">Enterprise</a>) are growing at <em>200% a year</em>. That speaks to the fact that we’ve recruited an incredibly talented team that has produced such fantastic results. </p>



<p>But, we have a lot of work ahead of us, and it’s going to take a different type of leader to get us through that work. </p>



<p>The type of people Stack Overflow serves has changed, and now, as a part of the developer ecosystem, we have a responsibility to create an online community that is far more diverse, inclusive, and welcoming of newcomers.</p>



<p>In the decade or so since Stack Overflow started, the number of people employed as software developers <a href="https://www.bls.gov/careeroutlook/2019/data-on-display/projections-evaluation.htm?view_full">grew by 64%</a> in the US alone. The field is going to keep growing everywhere in the world, and the demand for great software developers far outstrips supply. So a big challenge for Stack Overflow is welcoming those new developers into the fold. As I’ve <a href="https://www.joelonsoftware.com/2018/04/23/strange-and-maddening-rules/">written</a>:</p>



<blockquote class="wp-block-quote"><p>One thing I’m very concerned about, as we try to educate the next generation of developers, and, importantly, get more diversity and inclusiveness in that new generation, is what obstacles we’re putting up for people as they try to learn programming. In many ways Stack Overflow’s specific rules for what is permitted and what is not are obstacles, but an even bigger problem is rudeness, snark, or condescension that newcomers often see.</p><p>I care a lot about this. Being a developer gives you an unparalleled opportunity to <a href="https://www.joelonsoftware.com/2016/12/09/developers-are-writing-the-script-for-the-future/">write the script for the future</a>. All the flak that Stack Overflow throws in the face of newbies trying to become developers is actively harmful to people, to society, and to Stack Overflow itself, by driving away potential future contributors. And programming is hard enough; we should see our mission as making it easier.</p></blockquote>



<p>The world has started taking a closer look at tech, and understanding that software and the internet are not just tools; they are shaping the future of society. Big tech companies are struggling with their place in the world. Stack Overflow is situated at the right place to be influential in how that future develops, and that is going to take a new type of leader.</p>



<div class="wp-block-image"><figure class="alignright is-resized"><img loading="lazy" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?fit=730%2C973&amp;ssl=1" alt="" class="wp-image-3539" width="182" height="243" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?w=3024&amp;ssl=1 3024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?resize=225%2C300&amp;ssl=1 225w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?resize=768%2C1024&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?resize=1200%2C1600&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/03/IMG_5949.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 182px) 100vw, 182px" /><figcaption>new dog, too</figcaption></figure></div>



<p>It will not be easy to find a CEO who is the right person to lead that mission. We will, no doubt, hire one of those fancy executive headhunters to help us in the search. But, hey, this is Stack Overflow. If there’s one thing I have learned by now, it’s that there’s always someone in the community who can answer the questions I can’t. </p>



<p>So we decided to put this announcement out there in hopes of finding great candidates that might have been under the radar. We’re especially focused on identifying candidates from under-represented groups, and making sure that every candidate we consider is deeply committed to making our company and community more welcoming, diverse, and inclusive.</p>



<p>Over the years, Fog Creek Software created several incredible hits and many wonderful memories along the way. It is great to watch Trello (under Michael Pryor) and Glitch (under Anil Dash) growing into enormously valuable, successful, and influential products with dedicated leaders who took these products much further than I ever could have, and personally I’m excited to see where Stack Overflow can go and turn my attention to the next thing.<br></p>


guid: https://www.joelonsoftware.com/?p=3612
title: Welcome, Prashanth!
author: Joel Spolsky
url: https://www.joelonsoftware.com/2019/09/24/announcing-stack-overflows-new-ceo/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>Last March, I <a href="https://www.joelonsoftware.com/2019/03/28/the-next-ceo-of-stack-overflow/">shared</a> that we were starting to look for a new CEO for Stack Overflow. We were looking for that rare combination of someone who could foster the community while accelerating the growth of our businesses, especially Teams, where we are starting to close many huge deals and becoming a hyper-growth enterprise software company very quickly. This is not something I’m particularly good at, and I thought it was time to bring on more experienced leadership.</p>



<p>The Board of Directors nominated a search committee and we went through almost 200 candidates. It speaks to how well respected&nbsp; a company Stack Overflow is that we found ourselves in the rare position of having plenty of highly qualified executives who were excited about the opportunity. Nevertheless, one of them really stood out, and we are pleased to let you know that we have selected Prashanth Chandrasekar as our next CEO. His first day will be October 1st.</p>



<p>Prashanth was born in Bangalore, India, the city with the highest number of Stack Overflow users in the world, one of the global capitals for software developers writing the script for the future. He started out as a software engineer before moving over to management. He has a BS in Computer Engineering from the University of Maine, a Masters in Engineering Management from Cornell, and an MBA from Harvard. He worked at Capgemini as a management consultant and Barclays as an investment banker in their technology group before joining Rackspace in San Antonio, Texas.</p>



<p>At Rackspace, Prashanth really proved his mettle, creating from scratch a completely new business unit inside the company, the <a href="https://www.rackspace.com/cloud/public">Global Managed Public Clouds Business</a>. This group serves companies around the world who need help running on AWS, Azure, Google, and so on. Under his leadership, Rackspace successfully pivoted from a leading managed hosting company to a cloud services company. And he did this while working with developers both inside Rackspace and outside, so he understands our vision of “writing the script for the future” better than anyone I’ve met.</p>



<p>This is an exciting time for Stack Overflow, and we have some big goals for the year ahead. We want to make Stack Overflow more diverse, inclusive, and welcoming. And we want to make it possible for knowledge workers everywhere to use Stack Overflow to get answers to the proprietary questions that are specific to their organizations and teams. We’re doing great work and making great progress in these areas, and I’m confident that Prashanth has some great ideas about how to move forward faster on all our goals.</p>



<p>As you know, I’m keeping my job as Chairman of the Board, so I’ll continue to be closely involved. Being Stack Overflow’s CEO has been an honor, and I can’t wait to see the things the team accomplishes in the year ahead. This will be a great new chapter for Stack Overflow.<br></p>


guid: https://www.joelonsoftware.com/?p=3635
title: So, how’s that retirement thing going, anyway?
author: Joel Spolsky
url: https://www.joelonsoftware.com/2019/12/05/so-hows-that-retirement-thing-going-anyway/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>For the last couple of months, Prashanth Chandrasekar has been getting settled in as the <a href="https://www.joelonsoftware.com/2019/09/24/announcing-stack-overflows-new-ceo/">new CEO of Stack Overflow</a>. I&#8217;m still going on some customer calls and have a weekly meeting with him, but I have freed up a lot of time. I&#8217;m also really enjoying discovering just how little I knew about running medium-sized companies, as I watch Prashanth rearrange everything—for the better. It&#8217;s really satisfying to realize that the best possible outcome for me is if he proves what a bad CEO I was by doing a much better job running the company.</p>



<p>Even though I live in Manhattan&#8217;s premier NORC (&#8220;Naturally Occurring Retirement Community,&#8221;) I&#8217;m thinking of this time as a <a href="https://www.joelonsoftware.com/2000/03/18/more-on-sabbaticals/">sabbatical</a>, not retirement. And in fact I&#8217;m really, really busy, and, in the interest of deflecting a million questions about what I&#8217;m doing nowadays, thought I&#8217;d update my long-suffering readers here.</p>



<figure class="wp-block-image size-large"><img loading="lazy" width="2560" height="1920" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?fit=730%2C548&amp;ssl=1" alt="" class="wp-image-3636" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?w=2560&amp;ssl=1 2560w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=1536%2C1152&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=2048%2C1536&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_9590-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" /><figcaption>This adorable little fella, Cooper, is two. If your web app needs a mascot, apply within. </figcaption></figure>



<p>I&#8217;m chairman of three companies. You probably know all about Stack Overflow so I’ll skip ahead. </p>



<p>Fog Creek Software has been renamed <a href="https://glitch.com/">Glitch</a>, &#8220;the friendly community for building the web.&#8221; Under CEO <a href="http://anildash.com/">Anil Dash</a>, they have grown to <a href="https://www.theverge.com/2019/7/9/20683018/glitch-2-5-million-apps-code-remix-anil-dash">millions of apps</a> and raised a decent <a href="https://glitch.com/culture/glitch-raises-30m-series-a-round-from-tiger-global/">round of money</a> to accelerate that growth. I think that in every era there has to be some kind of simplified programming environment for the quiet majority of developers who don&#8217;t need fancy administration features for their code, like git branches or multistep deployment processes; they just want to write code and have it run. Glitch is aimed at those developers.</p>



<p>The third company, <a href="https://hash.ai/">HASH</a>, is still kind of under the radar right now, although today they put a whole bunch of words up on their website so I guess I can give you a preview. HASH is building an open source platform for doing simulations. It&#8217;s a great way to model problems where you have some idea of how every agent is supposed to behave, but you don&#8217;t really know what all that is going to add up to.</p>



<p>For example, suppose you&#8217;re a city planner and you want to model traffic so that you can make a case for a new bus line. You can, sort of, pretend that every bus takes 50 cars off the road, but that&#8217;s not going to work unless you can find 50 commuters who will all decide to take your new bus line&#8230; and the way they decide is that they check if the bus is actually going to save them time and money over just driving. This is a case where you can actually simulate the behavior of every &#8220;agent&#8221; in your model, like <a href="https://en.wikipedia.org/wiki/City-building_game">Cities: Skylines</a> does, and figure out the results. Then you can try thousands or millions of different potential bus routes and see which ones actually reduce traffic. </p>



<p>This kind of modeling is incredibly computationally intensive, but it works even when you don&#8217;t have a closed-form formula for how bus lines impact traffic, or, in general, how individual agents’ behavior affects overall outcomes. This kind of tool will be incredibly useful in far-ranging problems, like epidemiology, econometrics, urban planning, finance, political science, and a lot of other areas which are not really amenable to closed-form modeling or common &#8220;AI&#8221; techniques. (I love putting AI in “scare” “quotes”. There are a lot of startups out there trying to train machine learning models with way too little data. Sometimes the models they create just reproduce the bad decision making of the humans they are trained on. In many cases a model with simulated agents running a white box algorithm is going to be superior). </p>



<p>Ok, so those are the three companies I&#8217;m still working on in some way or another. That still leaves me with a couple of free days every week which I&#8217;m actually using to work on some electronics projects. </p>



<figure class="wp-block-image size-large"><img loading="lazy" width="2560" height="828" src="https://i1.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?fit=730%2C236&amp;ssl=1" alt="" class="wp-image-3639" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?w=2560&amp;ssl=1 2560w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=300%2C97&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=1024%2C331&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=768%2C249&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=1536%2C497&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=2048%2C663&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?resize=1200%2C388&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/IMG_7374-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" /></figure>



<p>In particular, I&#8217;m really into pixel-addressable RGB LEDs, like those WS2812b and APA102-type things. Right now I&#8217;m <a href="https://blinkylights.blog/">working</a> on designing a circuit board that connects a Teensy 3.2 controller, which can drive up to 4416 LEDs at a high frame rate, to a WizNET Ethernet adapter, and then creating some software which can be used to distribute 4416 pixels worth of data to each Teensy over a TCP-IP network in hopes of creating huge installations with hundreds of thousands of pixels. If that made any sense at all, you&#8217;re probably already a member of the <a href="https://www.facebook.com/groups/LEDSAREAWESOME/">LEDs ARE AWESOME</a> Facebook group and you probably think I&#8217;m dumb. If that doesn’t make any sense, rest assured that I am probably not going to burn down the apartment because I am very careful with the soldering iron almost every time. </p>



<figure class="wp-block-image"><img loading="lazy" width="730" height="548" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=730%2C548&#038;ssl=1" alt="" class="wp-image-3642" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?w=2560&amp;ssl=1 2560w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=1536%2C1152&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=2048%2C1536&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2019/12/img_7185-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></figure>



<p></p>


guid: https://www.joelonsoftware.com/?p=3701
title: HASH: a free, online platform for modeling the world
author: Joel Spolsky
url: https://www.joelonsoftware.com/2020/06/18/hash-a-free-online-platform-for-modeling-the-world/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>Sometimes when you&#8217;re trying to figure out the way the world works, basic math is enough to get you going. If we increase the hot water flow by <em>x</em>, the temperature of the mixture goes up by <em>y</em>. </p>



<p>Sometimes you&#8217;re working on something that&#8217;s just too complicated for that, and you can&#8217;t even begin to guess how the inputs affect the outputs. At the warehouse, everything seems to go fine when you have less than four employees, but when you hit five employees, they get in each others&#8217; way so much that the fifth employee effectively does no additional work.</p>



<figure class="wp-block-image size-large"><img loading="lazy" width="730" height="512" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2020/06/5ec6e31a68c106c99a6c6836.gif?resize=730%2C512&#038;ssl=1" alt="" class="wp-image-3702" data-recalc-dims="1"/></figure>



<p>You may not understand the relationship between the number of employees and the throughput of the warehouse, but you definitely know what everybody is doing. If you can imagine writing a little bit of JavaScript code to simulate the behavior of each of your workers, you can run a simulation and see what actually happens. You can tweak the parameters and the rules the employees follow to see how it would help, and you can really gain some traction understanding, and then solving, very complex problems.</p>



<p>That&#8217;s what <a href="https://hash.ai/">hash.ai</a> is all about. Read David&#8217;s <a href="https://hash.ai/about/mission">launch blog post</a>, then try building your own simulations!</p>


guid: https://www.joelonsoftware.com/?p=3810
title: Kinda a big announcement
author: Joel Spolsky
url: https://www.joelonsoftware.com/2021/06/02/kinda-a-big-announcement/
feedurl: file://./rss/VYPJXZwH.rss
enclosure_type: video/quicktime
content_mime_type: text/html
content:
<p>The other day I was talking to a young developer working on a code base with tons of COM code, and I told him that even before he was born, everyone knew that COM was already so deeply obsolete that it was impossible to find anyone who knew enough to work on it. And yet they still have this old COM code base, and they still have one old programmer holding onto their job by being the only human left on the planet with a brain big enough to manually manage multithreaded objects. I remember that COM was like Gödels Theorem: it seemed important, and you could understand it all long enough to pass an exam, but ultimately it is mostly just a demonstration of how far human intelligence can be made to stretch under extreme duress.</p>



<p>And, <em>bubbeleh</em>, if there is one thing we have learned, it&#8217;s that the things that make it <em>easier</em> on your brain are the things that matter.</p>



<div class="wp-block-image"><figure class="alignright size-large is-resized"><img loading="lazy" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1.jpg?resize=256%2C192&#038;ssl=1" alt="" class="wp-image-3822" width="256" height="192" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=1536%2C1152&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=2048%2C1536&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/63890965609__42347879-1BEE-4AD0-B24F-289733D844C1-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 256px) 100vw, 256px" data-recalc-dims="1" /></figure></div>



<p>Programming changes slowly. Really slowly.</p>



<p>Since I learned to code forty years ago, one thing that has mostly, <em>mostly</em>, changed about programming is that most developers no longer have to manage their own memory. Even getting that going took a long long time.</p>



<p>I took a few stupid years trying to be the CEO of a growing company during which I didn&#8217;t have time to code, and when I came back to web programming, after a break of about 10 years, I found Node, React, and other goodies, which are, don&#8217;t get me wrong, amazing? Really really great? But I also found that it took approximately the same amount of work to make a CRUD web app as it always has, and that there were some things (like handing a file upload, or centering) that were, shockingly, still just as randomly difficult as they were in VBScript twenty years ago. </p>



<p>Where are the flying cars?</p>



<p>The biggest problem is that developers of programming tools love to add things and hate to take things away. So things get harder and harder and more and more complex because there are more and more ways to do the same thing, each has pros and cons, and you are likely to spend as much time just figuring out which &#8220;rich text editor&#8221; to use as you are to implement it.</p>



<p>(Bill Gates, 1990: &#8220;How many f*cking programmers in this company are working on rich text editors?!&#8221;)</p>



<div class="wp-block-image"><figure class="alignright size-large is-resized"><img loading="lazy" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992.jpg?resize=256%2C192&#038;ssl=1" alt="" class="wp-image-3816" width="256" height="192" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=1536%2C1152&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=2048%2C1536&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/IMG_7992-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 256px) 100vw, 256px" data-recalc-dims="1" /></figure></div>



<p>So, in this world of slow, gradual change and alleged improvement, one thing did change literally overnight, or, to be precise, on September 15, 2008, which was when <a href="https://www.joelonsoftware.com/2008/09/15/stack-overflow-launches/">Stack Overflow launched</a>.</p>



<p>Six-to-eight weeks before that, Stack Overflow was only an idea. (*Actually Jeff started development in April). Six-to-eight weeks after that, it was a standard part of every developer&#8217;s toolkit: something they used every day. Something <em>had</em> changed about programming, and changed very fast: the way developers learned and got help and taught each other.</p>



<p>For many years, I was able to coast by telling delightful little stories about our incredible growth numbers, about the pay web site we made obsolete, and even about that one time when a Major Computer Book Publisher threatened to BURY US and launched their own Q&amp;A platform, which turned out to be more of a Scoff Generator than a Q&amp;A platform, but actually now almost anyone I talk to is too young to imagine The Days Before Stack Overflow, when the bookstore had an entire <em>wall</em> of Java and the way you picked a Rich Text Editor was going to Barnes and Noble and browsing through printed books for an hour, in the Rich Text Editor Component shelf.</p>



<p>Stack Overflow got to be pretty big as a business. The company grew faster than any individual&#8217;s skills at managing companies, especially mine, so a lot of the business team has changed over, and we now have a really world-class, experienced team that is doing much better than us founders. We&#8217;ve done incredible work building a recruiting platform for great developers, a &#8220;reach and relevance&#8221; platform for getting developers excited about your products, and, most importantly, <a href="https://stackoverflow.com/teams">Stack Overflow for Teams</a>, which is growing so quickly that soon every developer in the world will be using the power of Stack Overflow to get help with their own code base, not just common languages and libraries. </p>



<figure class="wp-block-image size-large"><img loading="lazy" width="730" height="548" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender.jpg?resize=730%2C548&#038;ssl=1" alt="" class="wp-image-3819" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=1024%2C768&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=300%2C225&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=768%2C576&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=1536%2C1152&amp;ssl=1 1536w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=2048%2C1536&amp;ssl=1 2048w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?resize=1200%2C900&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?w=1460&amp;ssl=1 1460w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/64308401762__1F48CB72-2FC2-4F18-9337-91AEB9E491C7.fullsizerender-scaled.jpg?w=2190&amp;ssl=1 2190w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></figure>



<p>And yeah, the one thing I made sure of was that everyone that came into the company understood exactly why Stack Overflow works, and what is important to the developers that it is by and for. So while we haven&#8217;t always been perfect, we have kept true to our mission, and the current leadership is just as committed to the vision of Stack Overflow as the founders are.</p>



<p>Today we&#8217;re pleased to announce that Stack Overflow is joining <a href="https://www.prosus.com">Prosus</a>. Prosus is an investment and holding company, which means that the most important part of this announcement is that Stack Overflow will continue to operate independently, with the exact same team in place that has been operating it, according to the exact same plan and the exact same business practices. Don&#8217;t expect to see major changes or awkward &#8220;synergies&#8221;. The business of Stack Overflow will continue to focus on Reach and Relevance, and Stack Overflow for Teams. The entire company is staying in place: we just have different owners now.</p>



<p>This is, in some ways, the best possible outcome. Stack Overflow stays independent. The company has plenty of cash on hand to expand and deliver more features and fix the old broken ones. Right now, the biggest gating factor to how fast we can do this is just how fast we can <a href="https://stackoverflow.com/company/work-here">hire excellent people</a>. </p>



<figure class="wp-block-video"><video controls src="https://www.joelonsoftware.com/wp-content/uploads/2021/06/63017985094__5783FDD0-E895-4A7F-BB5A-A1CD6B6C9DCF.mov"></video></figure>



<p>I&#8217;ve been out of the day-to-day for a while now. Together with David Wilkinson, I&#8217;m helping to build <a href="https://hash.ai">HASH</a>. HASH makes it easy to build powerful simulations and make better decisions. As we worked on that, we discovered that too much of the data that you might need to run simulations needs to be fixed up before you can use it. That&#8217;s because data is often published on the web, using page description languages that are more concerned with formatting and consumption by humans. They lack the structure to make the data they contain readily accessed programatically, so step one is miserable screen scraping and data cleanup. That&#8217;s where a lot of people give up.</p>



<div class="wp-block-image"><figure class="alignright size-large is-resized"><img loading="lazy" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864.jpg?resize=288%2C512&#038;ssl=1" alt="" class="wp-image-3820" width="288" height="512" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=576%2C1024&amp;ssl=1 576w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=169%2C300&amp;ssl=1 169w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=768%2C1365&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=864%2C1536&amp;ssl=1 864w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=1152%2C2048&amp;ssl=1 1152w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?resize=1200%2C2133&amp;ssl=1 1200w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2021/06/D88FF351-904A-4C36-94C7-0C27D9BEF864-scaled.jpg?w=1440&amp;ssl=1 1440w" sizes="(max-width: 288px) 100vw, 288px" data-recalc-dims="1" /></figure></div>



<p>We think we have an interesting way to fix this. If it works, we&#8217;ll change the web as quickly and completely as Stack Overflow changed programming. But it&#8217;s kind of ambitious and maybe a little too GRAND. If you are interested in joining that crazy journey, do reach out. The whole thing is going to be open source, so just hang on, and we&#8217;ll have something up on GitHub for you to play with.</p>



<p>See you soon!</p>



<p></p>

enclosure_url:
https://www.joelonsoftware.com/wp-content/uploads/2021/06/63017985094__5783FDD0-E895-4A7F-BB5A-A1CD6B6C9DCF.mov


guid: https://www.joelonsoftware.com/?p=3906
title: Making the web better. With blocks!
author: Joel Spolsky
url: https://www.joelonsoftware.com/2022/01/27/making-the-web-better-with-blocks/
feedurl: file://./rss/VYPJXZwH.rss
content_mime_type: text/html
content:
<p>You&#8217;ve probably seen web editors based on the idea of <em>blocks</em>. I&#8217;m typing this in WordPress, which has a little + button that brings up a long list of potential blocks that you can insert into this page:</p>



<figure class="wp-block-image size-large is-resized"><img src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/01/wordpressblocks.png?resize=260%2C573&#038;ssl=1" alt="" class="wp-image-3908" width="260" height="573" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/01/wordpressblocks.png?resize=465%2C1024&amp;ssl=1 465w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/01/wordpressblocks.png?resize=136%2C300&amp;ssl=1 136w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/01/wordpressblocks.png?w=676&amp;ssl=1 676w" sizes="(max-width: 260px) 100vw, 260px" data-recalc-dims="1" /></figure>



<p>This kind of &#8220;insert block&#8221; user interface concept is showing up in almost every blogging tool, web editor, note-taking app, and content management system. People like it and it makes sense. </p>



<p>We have seem to have standardized on one thing: the <strong>/</strong> key to insert a new block. Everything else, though, is completely proprietary and non-standard. </p>



<p>I thought, wouldn&#8217;t it be cool if blocks were interchangeable and reusable across the web?</p>



<p>Until now, every app that wants blocks has to implement them from scratch. Want a calendar block? Some kind of fancy Kanban board? Something to embed image galleries? Code it up yourself, buddy.</p>



<p>As a result of the non-standardization of blocks, our end-users suffer. If someone is using my blog engine, they can only use those blocks that I had time to implement. Those blocks may be pretty basic or incomplete. Users might want to use a fancier block that they saw in WordPress or Medium or Notion, but my editor doesn&#8217;t have it. Blocks can&#8217;t be shared or moved around very easily, and our users are limited to the features and capabilities that we had time to re-implement.</p>



<p>To fix this, we&#8217;re going to create a protocol called the <a href="https://blockprotocol.org">Block Protocol</a>. </p>



<p>It&#8217;s open, free, non-proprietary, we want it to be everywhere on the web. </p>



<p>It&#8217;s just a protocol that embedding applications can use to embed blocks. Any block can be used in any embedding application if they all follow the protocol.</p>



<p>Our hope is that this will make life much easier for app developers to support a huge variety of block types. At the same time, anyone can develop a block once and have it work in any blog platform, note-taking app, or content management system. It is all 100% free, open, and any sample code we develop showing how to use the protocol will be open-source.</p>



<p>We&#8217;ve released a very early draft of the Block Protocol, and we&#8217;ve started building some very simple blocks and a simple editor that can host them.</p>



<figure class="wp-block-image size-large is-style-default"><img loading="lazy" width="730" height="485" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_1-v2.png?resize=730%2C485&#038;ssl=1" alt="" class="wp-image-3923" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_1-v2.png?resize=1024%2C680&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_1-v2.png?resize=300%2C199&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_1-v2.png?resize=768%2C510&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_1-v2.png?w=1200&amp;ssl=1 1200w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></figure>



<p>We&#8217;re hoping to foster an open source community that creates a huge open source library of amazing blocks:</p>



<figure class="wp-block-image size-large is-style-default"><img loading="lazy" width="730" height="333" src="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_2-v2.png?resize=730%2C333&#038;ssl=1" alt="" class="wp-image-3924" srcset="https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_2-v2.png?resize=1024%2C467&amp;ssl=1 1024w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_2-v2.png?resize=300%2C137&amp;ssl=1 300w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_2-v2.png?resize=768%2C350&amp;ssl=1 768w, https://i0.wp.com/www.joelonsoftware.com/wp-content/uploads/2022/02/bp_illustration_2-v2.png?w=1200&amp;ssl=1 1200w" sizes="(max-width: 730px) 100vw, 730px" data-recalc-dims="1" /></figure>



<p>What can be a block? </p>



<ul><li>Anything that makes sense in a document: a paragraph, list, table, diagram, or a kanban board. </li><li>Anything that makes sense on the web: an order form, a calendar, a video.</li><li>Anything that lets you interact with structured or typed data: I&#8217;ll get to that in a minute.</li></ul>



<p>If you work on any kind of editor—be it a blogging tool, a note-taking app, a content management system, or anything like that—you should allow your users to embed blocks that conform to the Block Protocol. This way you can write the embedding code once and immediately make your editor able to embed a rich variety of block types with no extra work on your part.</p>



<p>If you work on any kind of custom data type that might make sense to embed in web pages, you should support the Block Protocol. That way anybody with a hosting application that supports the protocol can embed your custom data type.</p>



<p>Because it&#8217;s all 100% open, we hope that the Block Protocol will become a web standard and commonly used across the Internet.</p>



<p>That will mean that common block types, from paragraphs and lists to images and videos, will get better and better. But it will also mean that some esoteric block types will be embeddable anywhere. Want to create a block that shows the Great Circle routing for a flight between two airports? Write the code for the block once and it can be embedded anywhere. </p>



<p>Oh, and <strong>one more thing</strong>. Blocks can be highly structured, that is, they can have types. That means that they magically become machine-readable without screen scraping. For example, if you want to create an event block to represent an event on a calendar, you will be able to specify a <em>schema</em> that describes the event data type in a standard way. That way tools like calendars can instantly parse and understand web pages that contain your event block, reliably.</p>



<p>Over time, it will mean that anyone can easy publish complex, typed data sets on the web that are automatically machine-readable without extra work. (Have you ever seen one of those websites where there&#8217;s a link to &#8220;download the data set in .XLS format&#8221;? Yeah, say goodbye to that.)</p>



<p><strong>We&#8217;re going public with this very early in the development process</strong> because we need a lot of help! </p>



<p>Everything we have so far is version 0.1. It&#8217;s simple and not very good yet and going to need some iteration before it has the hope of truly being a useful web protocol. </p>



<p>This is an open protocol, free and non-proprietary, and it&#8217;s going to make the open web much better if widely adopted, so we need to start getting people involved early, giving us feedback, and building new things!</p>



<p><a href="https://blockprotocol.org">Go read more about the Block Protocol now!</a></p>


