Brett.is

Thinking about transactional email

Thu, 14 Sep 2017 00:00:00 +0000

I have recently started working on a new project, Mailchemy, for managing all of your transactional email integrations behind one easy to use API.

+ +

For awhile now I have been thinking about the complications of transactional email. +In the past I have found myself wanting to change email service providers, either for pricing or reliability reasons, but I’ve found myself stuck. +I had integrated my application so heavily into their product that making any kind of move away from their platform becomes difficult. +You can’t fault me for making this choice– email applications require deep integration in order to truly leverage all of the benefits of these services.

+ +

When integrating with an email service provider it would have been preferable to have abstracted out as much of the integration as possible, making it easier to swap out email providers. +Since we had not abstracted away the API calls in our code, we found ourselves having to track down every use of that specific API and update for it the new provider. +Given how tightly coupled sending transactional email is with our core application, these calls are spread throughout the code base and checking to be sure we had all usages removed and tested was both time consuming and risky. What if we missed a spot and weren’t sending an important message to our customers?

+ +

In addition to deep integration with an email service provider for when we send emails we were also storing a bunch of our email templates on the provider’s platform. +This was a nice feature to ensure we didn’t have to deploy a code change to update the copy in an email, enabling non-technical team members to make changes to our emails easily. +However, this nice feature also locked us into the provider further and made it harder for us to extract and move all of those templates to another platform. +To have more ownership of our email templates across platforms, we started moving some of the templates into our code base, but this introduced its own set of problems: shuffling templates around between providers and code is time consuming and means there is another part of the email pipeline you have to manage yourself. +We also lost the ability for our non-technical team members to update templates without a code change.

+ +

Since we were a small team, we couldn’t justify the loss of flexibility and increased effort that making these changes would incur– we were bootstrapped, we had more important things to work on. +So we were stuck with the initial email provider we picked.

+ +

This work problem got me thinking more about transactional email. +It seems most people follow a similar basic process:

+ +

Generate the email body from a template
Optimize the email (inline CSS, minify HTML, etc)
Send the email
Create analytics or reports about the emails being sent
Track open/click rates for emails +

+ +

There are a lot of really good transactional email providers, most of which offer some form of all of these steps. +However, they don’t all necessarily offer the same features as other platforms or the exact features or configurations that someone will want. +As I discussed above, sometimes you find yourself in a position where you don’t want to lock yourself into any given platform because of features so you abstract and manage the features you want anyways.

+ +

In software we talk a lot about DRY- don’t repeat yourself. +We use abstraction and open source to leverage the time of our community to not re-invent the wheel, so why aren’t we doing this more for transactional emails?

+ +

I have started working on a new project, Mailchemy, which aims to help simplify transactional email. +Configure and manage all of your favorite email integrations in one place and utilize with one simple API call.

+ +

Mailchemy is the product that I wish I had when I was configuring transactional email at previous companies. +By using Mailchemy to handle all of your email integrations and calls to your email provider your initial integration along with any changes are easier to manage. +Changing an email provider can happen in just a few clicks instead of the countless hours spent hunting down that one last usage.

+ +

Mailchemy allows users to define their own custom transactional email pipelines full of your favorite integrations and utilize the email provider of your choice. +You will no longer have to make a decision on which email provider to use based on the features they offer.

+ +

No support for Pug? +No problem, if Mailchemy supports it, you can use it.

+ +

Want to change email providers? +Take all of your favorite integrations with you.

+ +

As of the writing of this article Mailchemy is still under active development, please visit https://mailchemy.com/ for more information and to subscribe for product updates.

Managing Go dependencies with git-subtree

Wed, 03 Feb 2016 00:00:00 +0000

+ +

Recently I have decided to make the switch to using git-subtree for managing dependencies of my Go projects.

+ +

For a while now I have been searching for a good way to manage dependencies for my Go +projects. I think I have finally found a work flow that I really like that uses +git-subtree.

+ +

When I began investigating different ways to manage dependencies I had a few small goals or concepts I wanted to follow.

+ +

Keep it simple

+ +

I have always been drawn to the simplicity of Go and the tools that surround it. +I didn’t want to add a lot of overhead or complexity into my work flow when programming in Go.

+ +

Vendor dependencies

+ +

I decided right away that I wanted to vendor my dependencies, that is, where all of my dependencies +live under a top level vendor/ directory in each repository.

+ +

This also means that I wanted to use the GO15VENDOREXPERIMENT="1" flag.

+ +

Maintain the full source code of each dependency in each repository

+ +

The idea here is that each project will maintain the source code for each of its dependencies +instead of having a dependency manifest file, like package.json or Godeps.json, to manage the dependencies.

+ +

This was more of an acceptance than a decision. It wasn’t a hard requirement that +each repository maintains the full source code for each of its dependencies, but +I was willing to accept that as a by product of a good work flow.

+ +

In come git-subtree

+ +

When researching methods of managing dependencies with git, I came across a great article +from Atlassian, The power of Git subtree. +Which outlined how to use git-subtree for managing repository dependencies… exactly what I was looking for!

+ +

The main idea with git-subtree is that it is able to fetch a full repository and place +it inside of your repository. However, it differs from git-submodule because it does not +create a link/reference to a remote repository, instead it will fetch all the files from that +remote repository and place them under a directory in your repository and then treats them as +though they are part of your repository (there is no additional .git directory).

+ +

If you pair git-subtree with its --squash option, it will squash the remote repository +down to a single commit before pulling it into your repository.

+ +

As well, git-subtree has ability to issue a pull to update a child repository.

+ +

Lets just take a look at how using git-subtree would work.

+ +

Adding a new dependency

+ +

We want to add a new dependency, github.com/miekg/dns +to our project.

+ +

git subtree add --prefix vendor/github.com/miekg/dns https://github.com/miekg/dns.git master --squash
+

+ +

This command will pull in the full repository for github.com/miekg/dns at master to vendor/github.com/miekg/dns.

+ +

And that is it, git-subtree will have created two commits for you, one for the squash of github.com/miekg/dns +and another for adding it as a child repository.

+ +

Updating an existing dependency

+ +

If you want to then update github.com/miekg/dns you can just run the following:

+ +

git subtree pull --prefix vendor/github.com/miekg/dns https://github.com/miekg/dns.git master --squash
+

+ +

This command will again pull down the latest version of master from github.com/miekg/dns (assuming it has changed) +and create two commits for you.

+ +

Using tags/branches/commits

+ +

git-subtree also works with tags, branches, or commit hashes.

+ +

Say we want to pull in a specific version of github.com/brettlangdon/forge which uses tags to manage versions.

+ +

git subtree add --prefix vendor/github.com/brettlangdon/forge https://github.com/brettlangdon/forge.git v0.1.5 --squash
+

+ +

And then, if we want to update to a later version, v0.1.7, we can just run the following:

+ +

git subtree pull --prefix vendor/github.com/brettlangdon/forge https://github.com/brettlangdon/forge.git v0.1.7 --squash
+

+ +

Making it all easier

+ +

I really like using git-subtree, a lot, but the syntax is a little cumbersome. +The previous article I mentioned from Atlassian (here) +suggests adding in git aliases to make using git-subtree easier.

+ +

I decided to take this one step further and write a git command, git-vendor +to help manage subtree dependencies.

+ +

I won’t go into much details here since it is outlined in the repository as well as at https://brettlangdon.github.io/git-vendor/, +but the project’s goal was to make working with git-subtree easier for managing Go dependencies. +Mainly, to be able to add subtrees and give them a name, to be able to list all current subtrees, +and to be able to update a subtree by name rather than repo + prefix path.

+ +

Here is a quick preview:

+ +

$ git vendor add forge https://github.com/brettlangdon/forge v0.1.5
+$ git vendor list
+forge@v0.1.5:
+    name:   forge
+    dir:    vendor/github.com/brettlangdon/forge
+    repo:   https://github.com/brettlangdon/forge
+    ref:    v0.1.5
+    commit: 4c620b835a2617f3af91474875fc7dc84a7ea820
+$ git vendor update forge v0.1.7
+$ git vendor list
+forge@v0.1.7:
+    name:   forge
+    dir:    vendor/github.com/brettlangdon/forge
+    repo:   https://github.com/brettlangdon/forge
+    ref:    v0.1.7
+    commit: 0b2bf8e484ce01c15b87bbb170b0a18f25b446d9
+

+ +

Why not…

+ +

Godep/<package manager here>

+ +

I decided early on that I did not want to “deal” with a package manager unless I had to. +This is not to say that there is anything wrong with godep +or any of the other currently available package managers out there, I just wanted to keep +the work flow simple and as close to what Go supports with respect to vendored dependencies +as possible.

+ +

git-submodule

+ +

I have been asked why not git-submodule, and I think anyone that has had to work +with git-submodule will agree that it isn’t really the best option out there. +It isn’t as though it cannot get the job done, but the extra work flow needed +when working with them is a bit of a pain. Mostly when working on a project with +multiple contributors, or with contributors who are either not aware that the project +is using submodules or who has never worked with them before.

+ +

Something else?

+ +

This isn’t the end of my search, I will always be keeping a look out for new and +different ways to manage my dependencies. However, this is by far my favorite as of yet. +If anyone has any suggestions, please feel free to leave a comment.

Write code every day

Thu, 02 Jul 2015 00:00:00 +0000

Just like a poet or an athlete practicing code every day will only make you better.

+ +

Lately I have been trying to get into blogging more and any article I read always says, “you need to write every day”. +It doesn’t matter if what I write down gets published, but forming the habit of trying to write something every day +is what counts. The more I write the easier it will become, the more natural it will feel and the better I will get at it.

+ +

This really isn’t just true of writing or blogging, it is something that can be said of anything at all. Riding a bike, +playing basketball, reading, cooking or absolutely anything at all. The more you do it, the easier it will become and +the better you will get.

+ +

As the title of the post will allude you to, this is also true of programming. If you want to be really good at programming +you have to write code every day. The more code you write the easier it’ll be to write and the better you will be at programming. +Just like any other task I’ve listed in this article, trying to write code every day, even if you are used to it, can be really +hard to do and a really hard habit to keep.

+ +

“What should I write?” The answer to this question is going to be different for everyone, but it is the hurdle which +you must first overcome to work your way towards writing code every day. Usually people write code to solve problems +that they have, but not everyone has problems to solve. There is usually a chicken and the egg problem. You need to +write code to have coding problems, and you need to have coding problems to have something to write. So, where should +you start?

+ +

For myself, one of the things I like doing is to rewrite things that already exist. Sometimes it can be hard to come up with a +new and different idea or even a new approach to an existing idea. However, there are millions of existing projects out +there to copy. The idea I go for is to try and replicate the overall goal of the project, but in my own way. That might +mean writing it in a different language, or changing the API for it or just taking some wacky new approach to solving the same issue.

+ +

More times than not the above exercise leads me to a problem that I then can go off and solve. For example, a few weeks ago +I sat down and decided I wanted to write a web server in go (think nginx/apache). I knew going into the project I wanted +a really nice and easy to use configuration file to define the settings. So, I did what most people do these days I and +used json, but that didn’t really feel right to me. I then tried yaml, but yet again didn’t feel like what I wanted. I +probably could have used ini format and made custom rules for the keys and values, but again, this is hacky. This spawned +a new project in order to solve the problem I was having and ended up being forge, +which is a hand coded configuration file syntax and parser for go which ended up being a neat mix between json and nginx +configuration file syntax.

+ +

Anywho, enough of me trying to self promote projects. The main point is that by trying to replicate something that +already exists, without really trying to do anything new, I came up with an idea which spawned another project and +for at least a week (and continuing now) gave me a reason to write code every day. Not only did I write something +useful that I can now use in any future project of mine, I also learned something I did not know before. I learned +how to hand code a syntax parser in go.

+ +

Ultimately, try to take “coding every day” not as a challenge to write something useful every day, but to learn +something new every day. Learn part of a new language, a new framework, learn how to take something apart or put +it back together. Write code every day and learn something new every day. The more you do this, the more you will +learn and the better you will become.

+ +

Go forth and happy coding. :)

Forge configuration parser

Sat, 27 Jun 2015 00:00:00 +0000

+ +

An overview of how I wrote a configuration file format and parser.

+ +

Recently I have finished the initial work on a project, +forge, which is a +configuration file syntax and parser written in go. Recently I was working +on a project where I was trying to determine what configuration +language I wanted to use and whether I tested out +YAML or +JSON or +ini, nothing really felt +right. What I really wanted was a format similar to +nginx +but I couldn’t find any existing packages for go which supported this +syntax. A-ha, I smell an opportunity.

+ +

I have always been interested by programming languages, by their +design and implementation. I have always wanted to write my own +programming language, but since I have never had any formal education +around the subject I have always gone about it on my own. I bring it +up because this project has some similarities. You have a defined +syntax that gets parsed into some sort of intermediate format. The +part that is missing is where the intermediate format is then +translated into machine or byte code and actually executed. Since this +is just a configuration language, that is not necessary.

+ +

Project overview

+ +

You can see the repository for +forge for current usage and +documentation.

+ +

Forge syntax is a file which is made up of directives. There are 3 +kinds of directives:

+ +

settings: Which are in the form <KEY> = <VALUE>
sections: Which are used to group more directives <SECTION-NAME> { <DIRECTIVES> }
includes: Used to pull in settings from other forge config files include <FILENAME/GLOB>

+ +

Forge also supports various types of setting values:

+ +

string: key = "some value";
bool: key = true;
integer: key = 5;
float: key = 5.5;
null: key = null;
reference: key = some_section.key;

+ +

Most of these setting types are probably fairly self explanatory +except for reference. A reference in forge is a way to have the +value of one setting be a pointer to another setting. For example:

+ +

global = "value";
+some_section {
+  key = "some_section.value";
+  global_ref = global;
+  local_ref = .key;
+  ref_key = ref_section.ref_key;
+}
+ref_section {
+  ref_key = "hello";
+}
+

+ +

In this example we see 3 examples of references. A reference value +is one which is an identifier (global) possibly multiple identifiers separated +with a period (ref_section.ref_key) as well references can begin +with a perod (.key). Every reference which is not prefixed with a period +is resolved from the global section (most outer level). So in this +example a reference to global will point to the value of +"value" and ref_section.ref_key will point to the value of +"hello". A local reference is one which is prefixed with a period, +those are resolved starting from the current section that the +setting is defined in. So in this case, local_ref will point to +the value of "some_section.value".

+ +

That is a rough idea of how forge files are defined, so lets see a +quick example of how you can use it from go.

+ +

package main
+
+import (
+    "github.com/brettlangdon/forge"
+)
+
+func main() {
+    settings, _ := forge.ParseFile("example.cfg")
+    if settings.Exists("global") {
+    	value, _ := settings.GetString("global");
+    	fmt.Println(value);
+    }
+    settings.SetString("new_key", "new_value");
+
+    settingsMap := settings.ToMap();
+    fmt.Println(settingsMaps["new_key"]);
+
+    jsonBytes, _ := settings.ToJSON();
+    fmt.Println(string(jsonBytes));
+}
+

+ +

How it works

+ +

Lets dive in and take a quick look at the parts that make forge +capable of working.

+ +

Example config file:

+ +

# Top comment
+global = "value";
+section {
+  a_float = 50.67;
+  sub_section {
+    a_null = null;
+    a_bool = true;
+    a_reference = section.a_float;  # Gets replaced with `50.67`
+  }
+}
+

+ +

Basically what forge does is take a configuration file in defined +format and parses it into what is essentially a map[string]interface{}. +The code itself is comprised of two main parts, the tokenizer (or scanner) and the +parser. The tokenizer turns the raw source code (like above) into a stream of tokens. If +you printed the token representation of the code above, it could look like:

+ +

(COMMENT, "Top comment")
+(IDENTIFIER, "global")
+(EQUAL, "=")
+(STRING, "value")
+(SEMICOLON, ";"
+(IDENTIFIER, "section")
+(LBRACKET, "{")
+(IDENTIFIER, "a_float")
+(EQUAL, "=")
+(FLOAT, "50.67")
+(SEMICOLON, ";")
+....
+

+ +

Then the parser takes in this stream of tokens and tries to parse them based on some known +grammar. For example, a directive is in the form +<IDENTIFIER> <EQUAL> <VALUE> <SEMICOLON> (where <VALUE> can be +<STRING>, <BOOL>, <INTEGER>, <FLOAT>, <NULL>, +<REFERENCE>). When the parser sees <IDENTIFIER> it’ll look ahead +to the next token to try and match it to this rule, if it matches then +it knows to add this setting to the internal map[string]interface{} +for that identifier. If it doesn’t match anything then it has a syntax +error and will throw an exception.

+ +

The part that I think is interesting is that I opted to just write the +tokenizer and parser by hand rather than using a library that converts +a language grammar into a tokenizer (like flex/bison). I have done +this before and was inspired to do so after learning that that is how +the go programming language is written, you can see here +parser.go +(not a light read at 2500 lines). The +scanner.go +and +parser.go +might proof to be slightly easier reads for those who are interested.

+ +

Conclusion

+ +

There is just a brief overview of the project and just a slight dip +into the inner workings of it. I am extremely interested in continuing +to learn as much as I can about programming languages and +parsers/compilers. I am going to put together a series of blog posts +that walk through what I have learned so far and which might help +guide the reader through creating something similar to forge.

+ +

Enjoy.

What I'm up to these days

Fri, 19 Jun 2015 00:00:00 +0000

It has been awhile since I have written anything in my blog. Might as well get started +somewhere, like a brief summary of what I have been working on lately.

+ +

It has been far too long since I last wrote in this blog. I always have these aspirations +of writing all the time about all the things I am working on. The problem generally comes +back to me not feeling confident enough to write about anything I am working on. “Oh, a +post like that probably already exists”, “There are smarter people than me out there +writing about this, why bother”. It is an unfortunate feeling to try and get over.

+ +

So, here is where I am making an attempt. I will try to write more, it’ll be healthy for +me. I always hear of people setting reminders in their calendars to block off time to +write blog posts, even if they end up only writing a few sentences, which seems like a +great idea that I indent to try.

+ +

Ok, enough with the “I haven’t been feeling confident dribble”, on to what I actually have +been up to lately.

+ +

Since my last post I have a new job. I am now Senior Software Engineer at +underdog.io. We are a small early stage startup (4 employees, just +over a year old) that is in the hiring space. For candidates our site basically acts like +a common application to now over 150 venture backed startups in New York City or San +Francisco. In the short time I have been working there, I am very impressed and glad that +I took their offer. I work with some awesome and smart people and I am still learning a +lot, whether it is about coding or just trying to run a business.

+ +

I originally started to end this post by talking about a programming project I have been +working on, but it ended up being 4 times longer than the text above and have decided +instead to write a separate post about it. Apparently even though I have been writing +lately, I have a lot to say.

+ +

Thanks for bearing with this “I have to write something” post. I am not going to make a +promise that I am going to write more, because it is something that could easily fall +through, like it usually does… but I shall give it my all!

Javascript Documentation Generation

Tue, 03 Feb 2015 00:00:00 +0000

I have always been trying to find a good Javascript documentation generator and +I have never really been very happy with any that I have found. So I’ve decided +to just write my own, DocAST.

+ +

The problem I have always had with any documentation generators is they are +either hard to theme or are sometimes very strict with the way doc strings are +suppose to be written, making them potentially difficult to switch between +documentation generators if you had to. So for a fun exercise I’ve decided to +just try writting one myself, DocAST.

+ +

What is different about DocAST? I’ve seen a few documentation parsers which use +regular expressions to parse out the comment blocks, which works perfectly well, +except I’ve decided to have some fun and use +AST parsing to grab the +code blocks from the scripts. As well, DocAST doesn’t try to force you in to any +specific theme or display, instead it is used simply to extract documentation +from scripts. Lastly, DocAST, doesn’t use any specific documentation format for +signifying parameters, returns or exceptions, it will traverse the AST of the +code block to find them for you, so most of the time you just need to add a +simple block comment describing the function above it.

+ +

Lets just get to an example:

+ +

// script.js
+
+/*
+ * This is my super cool function that does all sorts of cool stuff
+ **/
+function superCool(arg1, arg2){
+    if(arg1 === arg2){
+        throw new Exception("arg1 and arg2 cant be the same");
+    }
+
+    var sum = arg1 + arg2;
+    return sum;
+}
+

+ +

$ docast extract ./script.js
+$ cat out.json
+

+ +

[
+    {
+        "name": "superCool",
+        "params": [
+            "arg1",
+            "arg2"
+        ],
+        "returns": [
+            "sum"
+        ],
+        "raises": [
+            "Exception"
+        ],
+        "doc": " This is my super cool function that does all sorts of cool stuff\n"
+    }
+]
+

+ +

For more information check out the github page for +DocAST.

+ +

The other benefit I have found with a documentation parser (something that just +extracts the documentation information as opposed to trying to build it) is that +you can get fun and creative with how you use the information parsed. For +example, I’ve had someone suggest creating your doc strings as +yaml. When you parse out the string just parse the yaml +to get an object which is then easy to pass on to jade +or some other templating engine to generate your documentation. If you want to +see an example of this, just check out the documentation for DocAST +https://github.com/brettlangdon/docast/blob/master/lib/index.js#L127 and the +code used to generate the docs at http://brettlangdon.github.io/docast/ +https://github.com/brettlangdon/docast/tree/master/docs

Python Redis Queue Workers

Tue, 14 Oct 2014 00:00:00 +0000

+ +

Learn an easy, distributed approach to processing jobs +from a Redis queue in Python.

+ +

Recently I started thinking about a new project. I want to write my own Continuous Integration (CI) +server. I know what you are thinking… “Why?!” and yes I agree, there are a bunch of good ones out +there now, I just want to do it. The first problem I came across was how to have distributed workers +to process the incoming builds for the CI server. I wanted something that was easy to start up on +multiple machines and that needed minimal configuration to get going.

+ +

The design is relatively simple, there is a main queue which jobs can be pulled from and a second queue +that each worker process pulls jobs into to denote processing. The main queue is meant as a list of things that +have to be processed where the processing queues is a list of pending jobs which are being processed by the +workers. For this example we will be using Redis lists since they support +the short feature list we require.

+ +

worker.py

+ +

Lets start with the worker process, the job of the worker is to simply grab a job from the queue and process it.

+ +

import redis
+
+
+def process(job_id, job_data):
+    print "Processing job id(%s) with data (%r)" % (job_id, job_data)
+
+
+def main(client, processing_queue, all_queue):
+    while True:
+        # try to fetch a job id from "<all_queue>:jobs"
+        # and push it to "<processing_queue>:jobs"
+        job_id = client.brpoplpush(all_queue, processing_queue)
+        if not job_id:
+            continue
+        # fetch the job data
+        job_data = client.hgetall("job:%s" % (job_id, ))
+        # process the job
+        process(job_id, job_data)
+        # cleanup the job information from redis
+        client.delete("job:%s" % (job_id, ))
+        client.lrem(process_queue, 1, job_id)
+
+
+if __name__ == "__main__":
+    import socket
+    import os
+
+    client = redis.StrictRedis()
+    try:
+        main(client, "processing:jobs", "all:jobs")
+    except KeyboardInterrupt:
+        pass
+

+ +

The above script does the following: +1. Try to fetch a job from the queue all:jobs pushing it to processing:jobs +2. Fetch the job data from a hash key with the name job:<job_id> +3. Process the job information +4. Remove the hash key job:<job_id> +5. Remove the job id from the queue processing:jobs

+ +

With this design we will always be able to determine how many jobs are currently queued for process +by looking at the list all:jobs and we will also know exactly how many jobs are being processed +by looking at the list processing:jobs which contains the list of job ids that all workers are +working on.

+ +

Also we are not tied down to running just 1 worker on 1 machine. With this design we can run multiple +worker processes on as many nodes as we want. As long as they all have access to the same Redis server. +There are a few limitations which are all seeded in Redis’ limits on lists, +but this should be good enough to get started.

+ +

There are a few other approaches that can be taken here as well. Instead of using a single processing queue +we could use a separate queue for each worker. Then we can look at which jobs are currently being processed +by each individual worker, this approach would also give us the opportunity to have the workers try to fetch +from the worker specific queue first before looking at all:jobs so we can either assign jobs to specific +workers or where the worker can recover from failed processing by starting with the last job it was working +on before failing.

+ +

qw

+ +

I have developed the library qw or (QueueWorker) to implement a similar +pattern to this, so if you are interested in playing around with this or to see a more developed implementation +please checkout the projects github page for more information.

Lets Make a Metrics Beacon

Sun, 22 Jun 2014 00:00:00 +0000

+ +

Recently I wrote a simple javascript metrics beacon +library. Let me show you what I came up with and how it works.

+ +

So, what do I mean by “javascript metrics beacon library”? Think +RUM (Real User Monitoring) or +Google Analytics, +it is a javascript library used to capture/aggregate metrics/data +from the client side and send that data to a server either in one +big batch or in small increments.

+ +

For those who do not like reading articles and just want the code you +can find the current state of my library on github: https://github.com/brettlangdon/sleuth

+ +

Before we get into anything technical, lets just take a quick look at an +example usage:

+ +

<script type="text/javascript" src="//raw.githubusercontent.com/brettlangdon/sleuth/master/sleuth.min.js"></script>
+<script type="text/javascript">
+Sleuth.init({
+  url: "/track",
+});
+
+// static tags to identify the browser/user
+// these are sent with each call to `url`
+Sleuth.tag('uid', userId);
+Sleuth.tag('productId', productId);
+Sleuth.tag('lang', navigator.language);
+
+// set some metrics to be sent with the next sync
+Sleuth.track('clicks', buttonClicks);
+Sleuth.track('images', imagesLoaded);
+
+// manually sync all data
+Sleuth.sendAllData();
+</script>
+

+ +

Alright, so lets cover a few concepts from above, tags, metrics and syncing.

+ +

Metrics

+ +

Metrics are simply data points to track for a given request. Good metrics to record +are things like load times, elements loaded on the page, time spent on the page, +number of times buttons are clicked or other user interactions with the page.

+ +

Syncing

+ +

Syncing refers to sending the data from the client to the server. I refer to it as +“syncing” since we want to try and aggregate as much data on the client side and send +fewer, but larger, requests rather than having to make a request to the server for +each metric we mean to track. We do not want to overload the Client if we mean to +track a lot of user interactions on the site.

+ +

How To Do It

+ +

Alright, enough of the simple examples/explanations, lets dig into the source a bit +to find out how to aggregate the data on the client side and how to sync that data +to the server.

+ +

Aggregating Data

+ +

Collecting the data we want to send to the server isn’t too bad. We are just going +to take any specific calls to Sleuth.track(key, value) and store either in +LocalStorage or in an object until we need +to sync. For example this is the track method of Sleuth:

+ +

Sleuth.prototype.track = function(key, value){
+  if(this.config.useLocalStorage && window.localStorage !== undefined){
+    window.localStorage.setItem('Sleuth:' + key, value);
+  } else {
+    this.data[key] = value;
+  }
+};
+

+ +

The only thing of note above is that it will fall back to storing in this.data +if LocalStorage is not available as well we are namespacing all data stored in +LocalStorage with the prefix “Sleuth:” to ensure there is no name collision with +anyone else using LocalStorage.

+ +

Also Sleuth will be kind enough to capture data from window.performance if it +is available and enabled (it is by default). And it simply grabs everything it can +to sync up to the server:

+ +

Sleuth.prototype.captureWindowPerformance = function(){
+  if(this.config.performance && window.performance !== undefined){
+    if(window.performance.timing !== undefined){
+      this.data.timing = window.performance.timing;
+    }
+    if(window.performance.navigation !== undefined){
+      this.data.navigation = {
+        redirectCount: window.performance.navigation.redirectCount,
+        type: window.performance.navigation.type,
+      };
+    }
+  }
+};
+

+ +

For an idea on what is store in window.performance.timing check out +Navigation Timing.

+ +

Syncing Data

+ +

Ok, so this is really the important part of this library. Collecting the data isn’t +hard. In fact, no one probably really needs a library to do that for them, when you +just as easily store a global object to aggregate the data. But why am I making a +“big deal” about syncing the data either? It really isn’t too hard when you can just +make a simple AJAX call using jQuery $.ajax(...) to ship up a JSON string to some +server side listener.

+ +

The approach I wanted to take was a little different, yes, by default Sleuth will +try to send the data using AJAX to a server side url “/track”, but what about when +the server which collects the data lives on another hostname? +CORS can be less than +fun to deal with, and rather than worrying about any domain security I just wanted +a method that can send the data from anywhere I want back to whatever server I want +regardless of where it lives. So, how? Simple, javascript pixels.

+ +

A javascript pixel is simply a script tag which is written to the page with +document.write whose src attribute points to the url that you want to make the +call to. The browser will then call that url without using AJAX just like it would +with a normal script tag loading javascript. For a more in-depth look at tracking +pixels you can read a previous article of mine: +Third Party Tracking Pixels.

+ +

The point of going with this method is that we get CORS-free GET requests from any +client to any server. But some people are probably thinking, “wait, a GET request +doesn’t help us send data from the client to server”? This is why we will encode +our JSON string of data for the url and simply send in the url as a query string +parameter. Enough talk, lets see what this looks like:

+ +

var encodeObject = function(data){
+  var query = [];
+  for(var key in data){
+    query.push(encodeURIComponent(key) + '=' + encodeURIComponent(data[key]));
+  };
+
+  return query.join('&');
+};
+
+var drop = function(url, data, tags){
+  // base64 encode( stringify(data) )
+  tags.d = window.btoa(JSON.stringify(data));
+
+  // these parameters are used for cache busting
+  tags.n = new Date().getTime();
+  tags.r = Math.random() * 99999999;
+
+  // make sure we url encode all parameters
+  url += '?' + encodeObject(tags);
+  document.write('<sc' + 'ript type="text/javascript" src="' + url + '"></scri' + 'pt>');
+};
+

+ +

That is basically it. We simply base64 encode a JSON string version of the data and send +as a query string parameter. There might be a few odd things that stand out above, mainly +url length limitations of base64 encoded JSON string, the “cache busting” and the weird +breaking up of the tag “script”. A safe url length limit to live under is around +2000 +to accommodate internet explorer, which from some very crude testing means each reqyest +can hold around 50 or so separate metrics each containing a string value. Cache busting +can be read about more in-depth in my article again about tracking pixels +(http://brett.is/writing/about/third-party-tracking-pixels/#cache-busting), but the short +version is, we add random numbers and the current timestamp the query string to ensure that +the browser or cdn or anyone in between doesn’t cache the request being made to the server, +this way you will not get any missed metrics calls. Lastly, breaking up the script tag +into “sc + ript” and “scri + pt” makes it harder for anyone blocking scripts from writing +script tags to detect that a script tag is being written to the DOM (also an img or +iframe tag could be used instead of a script tag).

+ +

Unload

+ +

How do we know when to send the data? If someone is trying to time and see how much time +someone is spending on each page or wants to make sure they are collecting as much data +as they want on the client side then you want to wait until the last second before +syncing the data to the server. By using LocalStorage to store the data you can ensure +that you will be able to access that data the next time you see that user, but who wants +to wait? And what if the user never comes back? I want my data now dammit!

+ +

Simple, lets bind an event to window.onunload! Woot, done… wait… why isn’t my data +being sent to me? Initially I was trying to use window.onunload to sync data back, but +found that it didn’t always work with pixel dropping, AJAX requests worked most of the time. +After some digging I found that with window.onunload I was hitting a race condition on +whether or not the DOM was still available or not, meaning I couldn’t use document.write +or even query the DOM on unload for more metrics to sync on window.onunload.

+ +

In come window.onbeforeunload to the rescue! For those who don’t know about it (I +didn’t before this project), window.onbeforeunload is exactly what it sounds like +an event that gets called before window.onunload which also happens before the DOM +gets unloaded. So you can reliably use it to write to the DOM (like the pixels) or +to query the DOM for any extra information you want to sync up.

+ +

Conclusion

+ +

So what do you think? There really isn’t too much to it is there? Especially since we +only covered the client side of the piece and haven’t touched on how to collect and +interpret this data on the server (maybe that’ll be a follow up post). Again this is mostly +a simple implementation of a RUM library, but hopefully it sparks an interest to build +one yourself or even just to give you some insight into how Google Analytics or other +RUM libraries collect/send data from the client.

+ +

I think this project that I undertook was neat because I do not always do client side +javascript and every time I do I tend to learn something pretty cool. In this case +learning the differences between window.onunload and window.onbeforeunload as well +as some of the cool things that are tracked by default in window.performance I +definitely urge people to check out the documentation on window.performance.

+ +

TODO

+ +

What is next for Sleuth? I am not sure yet, +I am thinking of implementing more ways of tracking data, like adding counter support, +rate limiting, automatic incremental data syncs. I am open to ideas of how other people +would use a library like this, so please leave a comment here or open an issue on the +projects github page with any thoughts you have.

+ +

Links

+ +

Goodbye Grunt, Hello Tend

Mon, 09 Jun 2014 00:00:00 +0000

+ +

Recently decided to give Grunt a try, which caused me to write my +own node.js build system.

+ +

For the longest time I had refused to move away from Makefiles +for Grunt or some other node.js build system. +But I finally gave in and decided to take an afternoon to give Grunt a go. +Initially it seemed promising, Grunt had a plugin for everything and ultimately +it supporting watching files and directories (the one feature I really wanted +for my make build setup).

+ +

I tried to move over a fairly simplistic Makefile that I already had written +into a Gruntfile. However, after about an hour (or more) of trying to get grunt +setup with grunt-cli and all the other +plugins installed and configured to do the right thing I realized that Grunt +wasn’t for me. I took a simple 10 (ish) line Makefile and turned it into a 40+ +line Gruntfile and it still didn’t seem to do exactly what I wanted. What I +had to reflect on was why should I spend all this time trying to learn how to +configure some convoluted plugins when I already known the correct commands to +execute? Then I realized what I really wanted wasn’t a new build system but +simply watch for a Makefile

+ +

I have attempted to get some form of watch working with a Makefile in the past, +but it usually involves using inotify and I’ve never gotten it working exactly +like how I wanted. So, I decided to start writing my own system, because, why not +spend more time on perfecting my build system. My requirements were fairly simple, +I wanted a way to watch a directory/files for changes and when they do simply run +a single command (ultimately make <target>), I wanted the ability to also run +long running processing like node server.js and restart them if certain files +have changed, and lastly unlike other watch based systems I have seen I wanted +a way to run a command as soon as I start up the watch program (so you dont have +to start the watching, then go open/save a newline change to a file to get it to +build for the first time).

+ +

What I came up with was tend. Which solves +mostly all of my needs, which was simply “watch for make”. So how do you use it?

+ +

Installation

+ +

npm install -g tend
+

+ +

Usage

+ +

Usage:
+  tend
+  tend <action>
+  tend [--restart] [--start] [--ignoreHidden] [--filter <filter>] [<dir> <command>]
+  tend (--help | --version)
+
+Options:
+  -h --help             Show this help text
+  -v --version          Show tend version information
+  -r --restart          If <command> is still running when there is a change, stop and re-run it
+  -i --ignoreHidden     Ignore changes to files which start with "."
+  -f --filter <filter>  Use <filter> regular expression to filter which files trigger the command
+  -s --start            Run <command> as soon as tend executes
+

+ +

Example CLI Usage

+ +

The following will watch for changes to any js files in the directory ./src/ +when any of them change or are added it will run uglifyjs to combine them into +a single file.

+ +

tend --ignoreHidden --filter "*.js" ./src "uglifyjs -o ./public/main.min.js ./src/*.js"
+

+ +

The following will run a long running process, starting it as soon as tend starts +and restarting the program whenever files in ./routes/ has changed.

+ +

tend --restart --start --filter "*.js" ./routes "node server.js"
+

+ +

Config File

+ +

Instead of running tend commands singly from the command line you can provide +tend with a .tendrc file of multiple directories/files to watch with commands +to run.

+ +

The following .tendrc file are setup to run the same commands as shown above.

+ +

; global settings
+ignoreHidden=true
+
+[js]
+filter=*.js
+directory=./src
+command=uglifyjs -o ./public/main.min.js ./src/*.js
+
+[app]
+filter=*.js
+directory=./routes
+command=node ./app/server.js
+restart=true
+start=true
+

+ +

You can then simply run tend without any arguments to have tend watch for +all changes configured in your .tendrc file.

+ +

Running:

+ +

tend
+

+ +

Will basically execute:

+ +

tend --ignoreHidden --filter "*.js" ./src "uglifyjs -o ./public/main.min.js ./src/*.js" \
+  & tend --restart --start --filter "*.js" ./routes "node server.js"
+

+ +

Along with running multiple targets at once, you can run specific targets from +a .tendrc file as well, tend <target>.

+ +

tend js
+

+ +

Will run the js target once.

+ +

tend --ignoreHidden --filter "*.js" ./src "uglifyjs -o ./public/main.min.js ./src/*.js"
+

+ +

With Make

+ +

If I haven’t beaten a dead horse enough, I am a Makefile kind of person and +that is exactly what I wanted to use this new tool with. So below is an example +of a Makefile and it’s corresponding .tendrc file.

+ +

js:
+    uglifyjs -o ./public/main.min.js ./src/*.js
+
+app:
+    node server.js
+
+.PHONY: js app
+

+ +

ignoreHidden=true
+
+[js]
+filter=*.js
+directory=./src
+command=make js
+
+[app]
+filter=*.js
+directory=./routes
+command=make app
+restart=true
+start=true
+

+ +

Conclusion

+ +

So that is mostly it. Nothing overly exciting and nothing really new here, just +another watch/build system written in node to add to the list. For the most part +this tool does exactly what I want for now, but if anyone has any ideas on how +to make this better or even any other better/easier tools which do similar things +please let me know, I am more than willing to continue maintaining this tool.

Sharing Data from PHP to JavaScript

Sun, 16 Mar 2014 00:00:00 +0000

+ +

A quick example of how I decided to share dynamic content from PHP with my JavaScript.

+ +

So the other day I was refactoring some of the client side code I was working on and +came across something like the following:

+ +

page.php

+ +

<html>
+...
+
+<script type="text/javascript">
+var modelTitle = "<?=$myModel->getTitle()?>";
+
+// do something with modelTitle
+</script>
+</html>
+

+ +

There isn’t really anything wrong here, in fact this seems to be a fairly common practice +(from the little research I did). So… whats the big deal? Why write an article about it?

+ +

My issue with the above is, what if the JavaScript gets fairly large (as mine was). The +ideal thing to do is to move the js into it’s own file, minify/compress it and serve it +from a CDN so it doesn’t effect page load time. But, now we have content that needs to be +added dynamically from the PHP script in order for the js to run. How do we solve it? The +approach that I took, which probably isn’t original at all, but I think neat enough to +share, was to let PHP make the data available to the script through window.data.

+ +

page.php

+ +

<html>
+...
+<?php
+$pageData = array(
+    'modelTitle' => $myModel->getTitle(),
+);
+?>
+<script type="text/javascript">
+window.data = <?=json_encode($pageData)?>;
+</script>
+<script type="text/javascript" src="//my-cdn.com/scripts/page-script.min.js"></script>
+</html>
+

+ +

page-script.js

+ +

// window.data.modelTitle is available for me to use
+console.log("My Model Title: " + window.data.modelTitle);
+

+ +

Nothing really fancy, shocking, new or different here, just passing data from PHP to js. +Something to note is that we have to have our PHP code set window.data before we load +our external script so that window.data will be available when the script loads. Which +this shouldn’t be too much of an issue since most web developers are used to putting all +of their script tags at the end of the page.

+ +

Some might wonder why I decided to use window.data, why not just set +var modelTitle = "<?=$myModel->getTitle()?>";? I think it is better to try and have a +convention for where the data from the page will come from. Having to rely on a bunch of +global variables being set isn’t really a safe way to write this. What if you overwrite +an existing variable or if some other script overwrites your data from the PHP script? +This is still a cause for concern with window.data, but at least you only have to keep +track of a single variable. As well, I think organizationally it is easier and more concise +to have window.data = <?=json_encode($pageData)?>; as opposed to:

+ +

var modelTitle = "<?=$myModel->getTitle()?>";
+var modelId = "<?=$myModel->getId()?>";
+var username = "<?=getCurrentUser()?>";
+...
+

+ +

I am sure there are other ways to do this sort of thing, like with AJAX or having an +initialization function that PHP calls with the correct variables it needs to pass, etc. +This was just what I came up with and the approach I decided to take.

+ +

If anyone has other methods of sharing dynamic content between PHP and js, please leave a +comment and let me know, I am curious as to what most other devs are doing to handle this.

Cookieless User Tracking

Sat, 30 Nov 2013 00:00:00 +0000

+ +

A look into various methods of online user tracking without cookies.

+ +

Over the past few months, in my free time, I have been researching various +methods for cookieless user tracking. I have a previous article that talks +on how to write a +tracking server +which uses cookies to follow people between requests. However, recently +browsers are beginning to disallow third party cookies by default which means +developers have to come up with other ways of tracking users.

+ +

Browser Fingerprinting

+ +

You can use client side javascript to generate a +browser fingerprint, +or, a unique identifier for a specific users browser (since that is what cookies +are actually tracking). Once you have the browser’s fingerprint you can then +send that id along with any other requests you make.

+ +

var user_id = generateBrowserFingerprint();
+document.write(
+    '<script type="text/javascript" src="/track/user/"' + user_id + '></ sc' + 'ript>'
+);
+

+ +

Local Storage

+ +

Newer browsers come equipped with a feature called +local storage +, which is used as a simple key-value store accessible through javascript. +So instead of relying on cookies as your persistent storage, you can store the +user id in local storage instead.

+ +

var user_id = localStorage.getItem("user_id");
+if(user_id == null){
+    user_id = generateNewId();
+    localStorage.setItem("user_id", user_id);
+}
+document.write(
+    '<script type="text/javascript" src="/track/user/"' + user_id + '></ sc' + 'ript>'
+);
+

+ +

This can also be combined with a browser fingerprinting library for generating +the new id.

+ +

ETag Header

+ +

There is a feature of HTTP requests called an +ETag Header +which can be exploited for the sake of user tracking. The way an ETag works is +when a request is made the server will respond with an ETag header with +a given value (usually it is an id for the requested document, or maybe a hash +of it), whenever the bowser then makes another request for that document it will +send an If-None-Match header with the value of ETag provided by the server +last time. The server can then make a decision as to whether or not new content +needs to be served based on the id/hash provided by the browser.

+ +

As you may have figured out, instead we can assign a unique user id as the ETag +header for a response, then when the browser makes a request for that page again +it will send us the user id.

+ +

This is useful, except for the fact that we can only provide a single id per +user per endpoint. For example, if I use the urls /track/user and +/collect/data there is no way for me to get the browser to send the same +If-None-Match header for both urls.

+ +

Example Server

+ +

from uuid import uuid4
+from wsgiref.simple_server import make_server
+
+
+def tracking_server(environ, start_response):
+    user_id = environ.get("HTTP_IF_NONE_MATCH")
+    if not user_id:
+        user_id = uuid4().hex
+
+    start_response("200 Ok", [
+        ("ETag", user_id),
+    ])
+    return [user_id]
+
+
+if __name__ == "__main__":
+    try:
+        httpd = make_server("", 8000, tracking_server)
+        print "Tracking Server Listening on Port 8000..."
+        httpd.serve_forever()
+    except KeyboardInterrupt:
+        print "Exiting..."
+

+ +

Redirect Caching

+ +

Redirect caching is similar in concept to the the ETag tracking method where +we rely on the browser cache to store the user id for us. With redirect caching +we have our tracking url /track/, when someone goes there we perform a 301 +redirect to /<user_id>/track. The users browser will then cache that 301 +redirect and the next time the user goes to /track it will just go to +/<user_id>/track instead.

+ +

Just like the ETag method we run into an issue where this method really only +works for a single endpoint url. We cannot use it for an end all be all for +tracking users across a site or multiple sites.

+ +

Example Server

+ +

from uuid import uuid4
+from wsgiref.simple_server import make_server
+
+
+def tracking_server(environ, start_response):
+    if environ["PATH_INFO"] == "/track":
+        start_response("301 Moved Permanently", [
+            ("Location", "/%s/track" % uuid4().hex),
+        ])
+    else:
+        start_response("200 Ok", [])
+    return [""]
+
+
+if __name__ == "__main__":
+    try:
+        httpd = make_server("", 8000, tracking_server)
+        print "Tracking Server Listening on Port 8000..."
+        httpd.serve_forever()
+    except KeyboardInterrupt:
+        print "Exiting..."
+

+ + + +

A project worth noting is Samy Kamkar’s +Evercookie +which uses standard cookies, flash objects, silverlight isolated storage, +web history, etags, web cache, local storage, global storage… and more +all at the same time to track users. This library exercises every possible +method for storing a user id which makes it a reliable method for ensuring +that the id is stored, but at the cost of being very intrusive and persistent.

+ +

Other Methods

+ +

I am sure there are other methods out there, these are just the few that I +decided to focus on. If anyone has any other methods or ideas please leave a comment.

+ +

References

+ +

My New Website

Sat, 16 Nov 2013 00:00:00 +0000

Why did I redo my website? +What makes it any better? +Why are there old posts that are missing?

+ +

I just wanted to write a quick post about my new site. +Some of you who are not familiar with my site might not notice the difference, +but trust me… it is different and for the better.

+ +

So what has changed? +For starters, I think the new design is a little simpler than the previous, +but more importantly it is not longer in Wordpress. +It is now maintained with Wintersmith, +which is a static site generator which is built in node.js and +usesJade templates and markdown.

+ +

Why is this better? +Well for started I think writing in markdown is a lot easier than using Wordpress. +It means I can use whatever text editor I want (emacs in this case) to write my +articles. As well, I no longer need to have PHP and MySQL setup in order to just +serve up silly static content like blog posts and a few images. +This also means I can keep my blog entirely in GitHub.

+ +

So far I am fairly happy with the move to Wintersmith, except having to move all my +current blog posts over to markdown, but I will slowly keep porting some over until +I have them all in markdown. So, please bear with me during the time of transition +as there may be a few posts missing when I initially publish this new site.

+ +

Check out my blog in GitHub, brett.is.

The Fastest Python JSON Library

Sun, 22 Sep 2013 00:00:00 +0000

My results from benchmarking a handfull of Python JSON parsing libraries.

+ +

Most who know me well know that I am usually not one for benchmarks. +Especially blindly posted benchmark results in blog posts (like how this one is going to be). +So, instead of trying to say that “this library is better than that library” or to try and convince you that you are going to end up with the same results as me. +Instead remember to take these results with a grain of salt. +You might end up with different results than me. +Take these results as interesting findings which help supplement your own experiments.

+ +

Ok, now that that diatribe is over with LETS GET TO THE COOL STUFF! +We use JSON for a bunch of stuff at work, whether it is a third party system that uses JSON to communicate or storing JSON blobs in the database. +We have done some naive benchmarking in the past and came to the conclusion that jsonlib2 is the library for us. +Well, I started a personal project that also uses JSON and I decided to revisit benchmarking Python JSON libraries to see if there are any “better” ones out there.

+ +

I ended up with the following libraries to test: +standard lib json, jsonlib2, simplejson, yajl (yet another json library) and lastly ujson (ultrajson). +For the test, I wanted to test parsing and serializing a large json blob, in this case, I simply took a snapshot of data from the Twitter API Console. +Ok, enough with this context b.s. lets see some code and some results.

+ +

import json
+import timeit
+
+# json data as a str
+json_data = open("./fixture.json").read()
+# json data as a list
+data = json.loads(json_data)
+
+number = 500
+repeat = 4
+print "Average run time over %s executions repeated %s times" % (number, repeat)
+
+# we still store the fastest run times here
+fastest_dumps = (None, -1)
+fastest_loads = (None, -1)
+
+for library in ("ujson", "simplejson", "jsonlib2", "json", "yajl"):
+    print "-" * 20
+    # thanks yajl for not setting __version__
+    exec("""
+try:
+    from %s import __version__
+except Exception:
+    __version__ = None
+         """ % library)
+    print "Library: %s" % library
+    # for jsonlib2 this is a tuple... thanks guys
+    print "Version: %s" % (__version__, )
+
+    # time to time json.dumps
+    timer = timeit.Timer(
+        "json.dumps(data)",
+        setup="""
+import %s as json
+data = %r
+              """ % (library, data)
+    )
+
+    total = sum(timer.repeat(repeat=repeat, number=number))
+    per_call = total / (number * repeat)
+    print "%s.dumps(data): %s (total) %s (per call)" % (library, total, per_call)
+    if fastest_dumps[1] == -1 or total > fastest_dumps[1]:
+        fastest_dumps = (library, total)
+
+    # time to time json.loads
+    timer = timeit.Timer(
+        "json.loads(data)",
+        setup="""
+import %s as json
+data = %r
+              """ % (library, json_data)
+    )
+    total = sum(timer.repeat(repeat=repeat, number=number))
+    per_call = total / (number * repeat)
+    print "%s.loads(data): %s (total) %s (per call)" % (library, total, per_call)
+    if fastest_loads[1] == -1 or total > fastest_loads[1]:
+       fastest_loads = (library, total)
+
+    print "-" * 20
+    print "Fastest dumps: %s %s (total)" % fastest_dumps
+    print "Fastest loads: %s %s (total)" % fastest_loads
+

+ +

Ok, we need to talk about this code for a second. +It really is not the cleanest code I have ever written. +We start off by loading the fixture json data as both the raw json text and parse it into a python list of objects. +Then for each of the libraries we want to test, we try to get their version information and finally we use timeit to test how long it takes to serialize the parsed fixture data into a JSON string and then we test parsing the JSON string of the fixture data into a list of objects. +And lastly, we store the name of the library with the fastest total run time for either “dumps” or “loads” and then at the end we print which was fastest.

+ +

Here are the results I got when running on my macbook pro:

+ +

Average run time over 500 executions repeated 4 times
+--------------------
+Library: ujson
+Version: 1.33
+ujson.dumps(data): 1.97361302376 (total) 0.000986806511879 (per call)
+ujson.loads(data): 2.05873394012 (total) 0.00102936697006 (per call)
+--------------------
+Library: simplejson
+Version: 3.3.0
+simplejson.dumps(data): 3.24183320999 (total) 0.001620916605 (per call)
+simplejson.loads(data): 2.20791387558 (total) 0.00110395693779 (per call)
+--------------------
+Library: jsonlib2
+Version: (1, 3, 10)
+jsonlib2.dumps(data): 2.211810112 (total) 0.001105905056 (per call)
+jsonlib2.loads(data): 2.55381131172 (total) 0.00127690565586 (per call)
+--------------------
+Library: json
+Version: 2.0.9
+json.dumps(data): 2.35674309731 (total) 0.00117837154865 (per call)
+json.loads(data): 5.23104810715 (total) 0.00261552405357 (per call)
+--------------------
+Library: yajl
+Version: None
+yajl.dumps(data): 2.85826969147 (total) 0.00142913484573 (per call)
+yajl.loads(data): 3.03867292404 (total) 0.00151933646202 (per call)
+--------------------
+Fastest dumps: ujson 1.97361302376 (total)
+Fastest loads: ujson 2.05873394012 (total)
+

+ +

So there we have it. +My tests show that ujson is the fastest python json library (when running on my mbp and when parsing or serializing a “large” json dataset).

+ +

I have added the test scripts, fixture data and results in this gist if anyone wants to run locally and post their results in the comments below. +I would be curious to see the results of others.

The Battle of the Caches

Thu, 01 Aug 2013 00:00:00 +0000

A co-worker and I set out to each build our own http proxy cache. +One of them was written in Go and the other as a C++ plugin for +Kyoto Tycoon.

+ +

So, I know what most people are thinking: “Not another cache benchmark post, +with skewed or biased results.” But luckily that is not what this post is about; +there are no opinionated graphs showing that my favorite caching system happens +to be better than all the other ones. Instead, this post is about why at work we +decided to write our own API caching system rather than use Varnish +(a tested, tried and true HTTP caching system).

+ +

Let us discuss the problem we have to solve. The system we have is a simple +request/response HTTP server that needs to have very low latency (a few +milliseconds, usually 2-3 on average) and we are adding a third-party HTTP API +call to almost every request that we see. I am sure some people see the issue +right away, any network call is going to add at least a half to a whole millisecond +to your processing time and that is if the two servers are in the same datacenter, +more if they are not. That is just network traffic, now we must rely on the +performance of the third-party API, hoping that they are able to maintain a +consistent response time under heavy load. If, in total, this third-party API call +is adding more than 2 milliseconds response time to each request that our system +is processing then that greatly reduces the capacity of our system.

+ +

THE SOLUTION! Lets use Varnish. This is the logical solution, lets put a caching +system in front of the API. The content we are requesting isn’t changing very often +(every few days, if that) and it can help speed up the added latency from the API +call. So, we tried this but had very little luck; no matter what we tried we could +not get Varnish to respond in under 2 milliseconds per request (which is a main +requirement of solution we were looking for). That means Varnish is out, the next +solution is to write our own caching system.

+ +

Now, before people start flooding the comments calling me a troll or yelling at me +for not trying this or that or some other thing, let me try to explain really why +we decided to write our own cache rather than spend extra days investing time into +Varnish or some other known HTTP cache. We have a fairly specific requirement from +our cache, very low and consistent latency. “Consistent” is the key word that really +matters to us. We decided fairly early on that getting no response on a cache miss +is better for our application than blocking and waiting for a response from the +proxy call. This is a very odd requirement and most HTTP caching systems do not +support it since it almost defeats their purpose (be “slow” 1-2 times so you can be +fast all the other times). As well, HTTP is not a requirement for us, that is, +from the cache to the API server HTTP must be used, but it is not a requirement +that our application calls to the cache using HTTP. Headers add extra bandwidth +and processing that are not required for our application.

+ +

So we decided that our ideal cache would have 3 main requirements: +1. Must have a consistent response time, returning nothing early over waiting for a proper response +2. Support the Memcached Protocol +3. Support TTLs on the cached data

+ +

This behavior works basically like so: Call to cache, if it is a cache miss, +return an empty response and queue the request to a background process to make the +call to the API server, every identical request coming in (until the proxy call +returns a result) will receive an empty response but not add the request to the +queue. As soon as the proxy call returns, update the cache and every identical call +coming in will yield the proper response. After a given TTL consider the data in +the cache to be old and re-fetch.

+ +

This was then seen as a challenge between a co-worker, +Dan Crosta, and myself to see who +can write the better/faster caching system with these requirements. His solution, +entitled “CacheOrBust”, was a +Kyoto Tycoon plugin +written in C++ which simply used a subset of the memcached protocol as well as some +background workers and a request queue to perform the fetching. My solution, +Ferrite, is a +custom server written in Go +(originally written in C) that has the same functionality (except using +goroutines +rather than background workers and a queue). Both servers used +Kyoto Cabinet +as the underlying caching data structure.

+ +

So… results already! As with most fairly competitive competitions it is always a +sad day when there is a tie. Thats right, two similar solutions, written in two +different programming languages yielded similar results (we probably have +Kyoto Cabinet to thank). Both of our caching systems were able to yield us the +results we wanted, consistent sub-millisecond response times, averaging about +.5-.6 millisecond responses (different physical servers, but same datacenter), +regardless of whether the response was a cache hit or a cache miss. Usually the +morale of the story is: “do not re-invent the wheel, use something that already +exists that does what you want,” but realistically sometimes this isn’t an option. +Sometimes you have to bend the rules a little to get exactly what your application +needs, especially when dealing with low latency systems, every millisecond counts. +Just be smart about the decisions you make and make sure you have sound +justification for them, especially if you decide to build it yourself.

Browser Fingerprinting

Wed, 05 Jun 2013 00:00:00 +0000

+ +

Ever want to know what browser fingerprinting is or how it is done?

+ +

What is Browser Fingerprinting?

+ +

A browser or device fingerprint +is a term used to describe an identifier generated from information retrieved from +a single given device that can be used to identify that single device only. +For example, as you will see below, browser fingerprinting can be used to generate +an identifier for the browser you are currently viewing this website with. +Regardless of you clearing your cookies (which is how most third party companies +track your browser) the identifier should be the same every time it is generated +for your specific device/browser. A browser fingerprint is usually generated from +the browsers user agent, +timezone offset, list of installed plugins, available fonts, screen resolution, +language and more. The EFF did +a study +on how unique a browser fingerprint for a given client can be and which browser +information provides the most entropy. To see how unique your browser is please +check out their demo application +Panopticlick.

+ +

What can it used for?

+ +

Ok, so great, but who cares? How can browser fingerprinting be used? Right now +the majority of user tracking +is done by the use of cookies. For example, when you go to a website that has +tracking pixels +(which are “invisible” scripts or images loaded in the background of the web page) +the third party company receiving these tracking calls will inject a cookie into +your browser which has a unique, usually randomly generated, identifier that is +used to associate stored data about you like collected +site or search retargeting +data. This way when you visit them again with the same cookie they can lookup +previously associated data for you.

+ +

So, if this is how it is usually done why do we care about browser fingerprints? +Well, the main problem with cookies is they can be volatile, if you manually delete +your cookies then the company that put that cookie there loses all association with +you and any data they have on your is no longer useful. As well, if a client does +not allow third party cookies (or any cookies) on their browser then the company +will be unable to track the client at all.

+ +

A browser fingerprint on the other hand is a more constant way to identify a given +client, as long as they have javascript enabled (which seems to be a thing which +most websites cannot properly function without), which allows the client to be +identified even if they do not allow cookies for their browser.

+ +

How do we do it?

+ +

Like I mentioned before to generate a browser fingerprint you must have javascript +enabled as it is the easiest way to gather the most information about a browser. +Javascript gives us access to things like your screen size, language, installed +plugins, user agent, timezone offset, and other points of interest. This +information is basically smooshed together in a string and then hashed to generate +the identifier, the more information you can gather about a single browser the more +unique of a fingerprint you can generate and the less collision you will have.

+ +

Collision? Yes, if you end up with two laptops each of the same make, model, year, +os version, browser version with the exact same features and plugins enabled then +the hashes will be the exact same and anyone relying on their fingerprint will +treat both of those devices as the same. But, if you read the white paper by EFF +listed above then you will see that their method for generating browser fingerprints +is usually unique for almost 3 million different devices. There may be some cases +for companies where that much uniqueness is more than enough to use and rely on +fingerprints to identify devices and others where they have more than 3 +million users.

+ +

Where does this really come into play? Most websites usually have their users +create and account and log in before allowing them access to portions of the site or +to be able to lookup stored information, maybe their credit card payment +information, home address, e-mail address, etc. Where browser fingerprints are +useful is for trying to identify anonymous visitors to a web application. For +example, third party trackers +who are collecting search or other kinds of data.

+ +

Some Code

+ +

Their is a project on github +by user Valentin Vasilyev (Valve) +called fingerprintjs +which is a client side javascript library for generating browser fingerprints. +If you are interested in seeing some production worthy code of how to generate +browser fingerprints please take a look at that project, it uses information like +useragent, language, color depth, timezone offset, whether session or local storage +is available, a listing of all installed plugins and it hashes everything using +murmurhash3.

+ +

Your fingerprintjs Fingerprint: Could not generate fingerprint

+ + + + +

Resources: +* panopticlick.eff.org - find out how rare your browser fingerprint is. +* github.com/Valve/fingerprintjs - client side browser fingerprinting library.

Making Citizen’s Issues Accessible

DemocracyLab — Tue, 09 Aug 2022 22:36:33 GMT

Many citizens are not in the know of the happenings within their own community. This could be a lack of resources, public transport, or even time. The Council Data Project in Missoula, Montana, is a new and innovative way to change this and allows for more citizen engagement tools.

In about 20 hours, Smai Fullerton used pre-existing technology to install an instance of the Council Data Project for the city of Missoula, Montana. She did this through the encouragement and backing of Open Montana, a new program of DemocracyLab. This project records and digitizes city council meetings. In addition it allows individuals to search key terms from the transcript of the meeting themselves so they can find the information that has the most impact on them.

Search results for City Council Meetings about zoning

This project allows for more citizen engagement because citizens can stay informed whether people are able to go to the city council meeting or not. When asked the question, “What is the easiest way to get this project all across the country?”, Smai Fullerton said we need a great story, more immediate use cases, and the ability to provide examples of its utility.

This technology can also be developed further to be more user friendly and allow for a greater impact. The use of voice recognition to be able to search for specific city council members would be a useful addition so citizens are able to see specific opinions from their community leaders. Some other useful additions to the Council Data Project are: seeing how many times people are interrupted or interrupt someone, a minute count of how much time a member of the city council was speaking, and ability to search for keywords/phrases. The latter of these potential improvements could be the most useful. Vernacular used in a city council meeting versus the way individuals talk to each other and various professional settings are vastly different. Therefore, allowing for a related word search would likely help improve the accessibility for individuals.

Search results for events about zoning

Key information to highlight regarding the Council Data Project is its relation with Open Montana which is the maintainer for the Missoula Council Data Project. Though this technology is in more locations than Missoula it is also the same basic infrastructure. This means the “data processing pipelines, the web frontend, and data storage systems all share the same infrastructure specification.”

When discussing the features Council Data Project is planning on creating and implementing, Eva Brown had the following to say: “We have a lot of features in development right now. A whole page just for tracking legislation, an audio-based speaker classification model for annotated speaker names in the produced transcripts, a notifications feature that would work a bit like Google Search Alerts (i.e. "a meeting from June 7 discussed 'missing middle housing'"), and some more that are just getting started.”

All these improvements, though important, are more likely to be seen as part of a long-term plan for this technology and the City of Missoula. The technology of the Council Data Project is very important and with time we expect to see more citizen engagement and will be able to gain personal testimony on how this technology has been useful, and through this, be able to implement it in other cities around the country.

A Better Tech-a-thon: ground-breaking collaboration on Public Interest Technology

DemocracyLab — Thu, 11 Nov 2021 15:05:48 GMT

Students, faculty, and tech professionals from across the United States recently got together to collaborate on innovative tech-for-good projects as part of A Better Tech-a-thon.

The hands-on virtual event took place on October 14th and 15th, 2021, and was presented by DemocracyLab, Code for America, and New York University, with sponsorship from Amazon.

It was part of the A Better Tech Public Interest Technology Convention and Career Fair — the first event of its kind in the US to bring together talented students, leading researchers, organizations, and companies passionate about serving public interest through responsible and accountable tech. The event was convened by New York University (NYU), with support from New America’s Public Interest Technology University Network.

The goal of A Better Tech-a-thon was to provide participants an opportunity to hone their skills while networking and making valuable contributions to numerous tech-for-good projects. Projects were sourced from the Code for America Brigade Network and DemocracyLab and ranged in missions from educating children about the trees around them, to making it easier to locate food donation sites.

Hana Schank, co-director of the Public Interest Technology program at New America and author of Power to the Public, said that looking at the participating projects was like “taking a spin through people’s hopes and dreams for a better world.”

“The world that you are all envisioning is a beautiful place where technology is used intelligently and thoughtfully to connect people instead of dividing them; to empower people, especially those who have been historically marginalized; and to make not only government, but society work better and more effectively, ”Schank added.

Matt Statler, the Richman Family Director of Business Ethics and Social Impact Programming at NYU Stern said about the hackathon: “To see and hear all of these examples of incredible work and interesting challenges people have selected, public interest challenges that people are bringing technology to bear on, and a clearly vibrant and ongoing network for really great collaborations across geographies and disciplines… and the knowledge of civic infrastructure, government, and political science is really, really super.”

Mona Sloane, Director of the This is Not a Drill (TINAD) program and Future Imagination Collaboratory (FIC) Fellow at the NYU Tisch School of the Arts, applauded the young talented participants for choosing to get involved in public interest technology careers.

“I firmly believe in putting things on the road and translation work between disciplines, between the public, policymakers, technologists, and so on,” said Sloane.

She added, “I really do think that the hackathon work that has happened as part of this event, as well as the other hackathons run by DemocracyLab, are really important in practicing that and getting us to a place where we more routinely can do that work.”

Participating universities included City University of New York; Columbia University; New York University; Pepperdine University; Stanford University; University of California, Berkeley; and the University of Washington.

Collaborating on public interest projects

The exciting two-day hackathon virtually brought together coders, designers, researchers, project managers, subject matter experts, and test users from different time zones to work on 17 unique projects.

For students, the event provided a great opportunity to sharpen their skills, extend their networks, and begin long-term relationships as project contributors.

More experienced tech professionals were also able to further their passions by helping tech-for-good projects make incremental progress toward their long-term goals, as well as work with the next generation of talent.

The participating projects were:

59Boards — Aims to update or rebuild New York City’s system of 59 community board districts to allow the public to better get involved with local politics.
Accessibility Ratings — A crowd-sourced application for accessibility ratings that will help users and organizations adopt and use more accessible applications.
Banana App — A location-based food donation app system that connects food donors and low-income clients.
Council Data Project — An entirely free and open application that makes it easier to follow one’s local municipality’s council action.
EnCiv — "A portal for productive democracy online, using processes derived from in-person dialog & deliberation, because social media is like a mob!
Full Disclosure Project — Aims to disrupt the culture of secrecy that systematically and pervasively shields law enforcement misconduct.”
iSeaTree — A game-play app that helps kids learn, identify, and record trees in the U.S. and Canada and works with US Forest Service’s iTree CO2 calculator.
Neighborhood.org — Model.earth Lifecycle Tools for Schools and Communities using US EPA widgets.
OutreachApp — An app that streamlines communication between outreach groups providing services and resources to the unsheltered community
P4H — An app that empowers education systems in developing countries by facilitating collaboration and certifications for teachers and P4h staff.
Pipeline — Fighting unemployment & streamlining the job search by offering a centralized application tracking system and recruiting insights.
Proof of Humanity — Many immigrants are undocumented and don't get social services. BrightID approaches this issue with a "Proof-of humanity" tech we'll test.
Saving Political Sites — Local political candidate websites disappear post-elections -- this project collects and archives those websites to make them accessible.
Smart City Planning — OSM2GMNS aims to enable rapid creation of routable multimodal networks and demand in General Modeling Network Specification.
Transpare-NC — Starting with campaign finance data, TRANSPARE-NC plans to offer unprecedented access and insight into North Carolina politics.
Vot-ER — An organization at the crossroads of health and democracy that helps patients register to vote in healthcare settings.
Yellow Box Disaster Relief — Rapid and local coms and supply chain app for volunteers.

Get involved

Since launching its open-source platform in 2018, DemocracyLab has convened more than 20 tech-for-good hackathons aimed at supporting public interest technology.

The hackathons have allowed projects to execute a narrow scope of work and make incremental progress towards their long term goals, and volunteers to begin long-term relationships with projects that match their skills and interests while upskilling and advancing their careers.

For more information on how you can participate in DemocracyLab’s next tech-for-good event, please visit our events page.

For a list of ongoing volunteer opportunities, make sure to check out DemocracyLab’s project listings.

And if you're a company representative seeking innovative ways to increase employee enthusiasm and retention, take a look at our corporate engagement page.

Navigating Career Changes with Chocolate Milk Diplomacy

DemocracyLab — Wed, 08 Sep 2021 19:34:39 GMT

Searching for a job or navigating a career change can feel ambiguous and overwhelming. That’s why the DemocracyLab marketing team, in partnership with Chocolate Milk Diplomacy has created a 50-minute course to help make this process engaging, with clear and actionable steps. It will help you get started and point you to the resources you need to navigate your career transition.

ABOUT THE COURSE CREATORS

This course was created from the “Career Advancement Workshop” that was organized in partnership with DemocracyLab and led by Alex Berry of Chocolate Milk Diplomacy, an organization that provides career coaching, navigation, and development resources to marginalized and underrepresented communities.

COURSE SUMMARY

Here’s a quick summary of each of the four modules in the course.

Module 1 - CMD believes that storytelling is at the heart of career development. The first module introduces CMD’s Five Step Career Transition Process and walks you through how to identify your own personal career story.

Module 2 - A discussion of why career transitions are often so painful. In this module Berry offers some insights about how to move past some of the fears and confusion that can keep us from making progress.

Module 3 - Progress can be tough if you don't have a goal in mind. The third module focuses on how to identify where you want to go in your career. Berry walks through three categories to consider when deciding on a career that will fit holistically into your life.

Module 4 - Now that you have your career story and your career path, Berry gets into the details of how to identify and create action plans for filling in the gaps in your resume.

MORE RESOURCES

This four-part course will give you a great kickstart to your making your career advancement or career transition goals a reality. For more resources check out the media library on the Chocolate Milk Diplomacy website. Here you'll find workshops, podcasts, one-on-one consulting, and self-led workbooks.

And consider volunteering with DemocracyLab to do good while building your resume and your network. Check out our projects page to find something that interests you today.

Building a Database to End School Violence

DemocracyLab — Wed, 11 Aug 2021 21:29:41 GMT

Gun violence in schools is a public health issue affected almost 200,000 students firsthand between the 1999 shooting in Columbine, Colorado and 2018. The following year in 2019, there was an average of one school shooting every school week. But despite the frequency of such tragedies, nothing was done at scale to analyze data from the events to identify patterns and help prevent further catastrophe.

Seeing the forest through the trees

Professor and technology advocate Paul Privateer believes that technology is an intricate part of our social consciousness. In under a year, the world developed and distributed a COVID-19 vaccine, and people immediately went to work building aggregate search tools that allow anyone to find nearby doses. Why, wonders Professor Privateer, do we as a society not have the same response to school violence? With the right resources, parents or teachers could monitor behavioral changes in an individual, correlate them the location and demographics of a school, and see in an instant whether those data predict violent actions. With enough time to intervene, we could potentially prevent more tragedies from happening.

An idea born of tragedy

Throughout his career Paul Privateer has been a professor of technological sciences and digital culture at San Jose State, Arizona State University, Georgia Tech, and MIT. The idea for the Athena Project was born during his tenure at Arizona State University, after two graduate students committed suicide within a few months of each other. As Paul grieved the loss of these young people along with his community, he asked his colleagues to brainstorm ways that they could use technology to understand and analyze the psychological and social factors of violence towards oneself or others. Unable to get federal funding, they decided to proceed on their own. From this starting point Professor Privateer founded his Seattle-based non-profit NoSchoolViolence.org in 2017.

The Athena Project

The Athena project under the NoSchoolViolence umbrella is developing a national school violence database that’s searchable via a user-friendly app. Paul found that data about incidents of school violence were recorded in over 40 separate, local systems in narrative “trend reports.” Using powerful data science software, the team has been able to pull reports from the disparate sites housing the information, run comparative analyses to identify common data points, and organize that data into tables.

The long term goal of this project is to provide the Athena app as a resource for parents, school faculty and administrations, researchers, health professionals, state education departments, and state and federal agencies. The app will help them understand which combinations of behaviors and traits correlate to violence, and will offer tested and effective intervention methods. Through this analysis users will be able to build and sustain successful violence prevention programs.

Current state and next steps

Paul has attended 10 DemocracyLab hack-a-thons where he connected with volunteers skilled in wireframe design, search, machine learning, and natural language processing. At each event, Paul and the volunteers worked to advance the quality of the project’s dataset and the efficiency of its search models. By participating in these events Paul has discovered what improvements will benefit the project most and moved the Athena project forward iteratively. Once the backend data is fully converted, Paul will focus on refining the app's search models.

Get Involved

Currently, this project is still in the development phase, and there are lots of ways to get involved. If you’re a designer, data savvy, or specialize in machine learning, and you want to help create the first national school violence database and search tool this is a great project for you.

The Athena project is looking for volunteers with the following skills:

Machine learning experience
Working knowledge of search engines
Natural language processing
Data Conversion for structured queries

If you're not yet registered, registering with DemocracyLab is free and easy.

DemocracyLab is a 501(c)(3) nonprofit organization with a mission to empower people who use technology to advance the public good. We are headquartered in Seattle, Washington, USA.

Volunteers make the 2021 Summer Create-a-thon a Success

DemocracyLab — Wed, 11 Aug 2021 21:19:12 GMT

On July 10, 2021, DemocracyLab collaborated with Code for America Brigade Network for its Summer Create-a-thon. Ten projects came together with over 60 volunteers to make progress building tech-for-good. There was a diverse mix of causes that benefited, including environmental conservation, civic infrastructure, education, and social services. Each project lead set progress goals and worked with its volunteers to advance their technologies.

Who participated

Volunteers joined in from around the world to contribute their talents and skills to help our participating projects make critical developments. Both first-time volunteers and veterans were drawn by the opportunity to practice or learn new skills in coding, design, research, and project management while addressing concrete civic issues.

By the end of the day our participants felt like they’d made substantial progress toward the goals they’d set, and many volunteers intended to continue contributing to the project they’d worked with that day.

What was accomplished

Project leads came with a variety of tasks and goals. Many projects worked on building new prototypes and developing or expanding new versions of apps, websites, and tools. Volunteers had the opportunity to work in UX design, data analysis, content development, and both frontend and backend development. Some key achievements included building app prototypes, developing or expanding new versions of apps, website development, and onboarding new team members.

Here’s a list of the projects:

A/B Street–software that runs traffic simulations to find ways to make cities less reliant on cars–worked on drafting new blog posts for recent proposals for local road changes.
Accessibility Ratings–a crowdsourced app that helps users ensure they adopt accessible applications–worked on building prototypes for some newly completed designs using search services and parsing metadata in app stores.
CouncilDataProject–an app providing users a way to follow local municipality council activities–worked to expand their coverage to new cities for version 3 launch, including backend web scraping and frontend UX design.
EnCiv (Unpoll)–software crowdsourcing political debate questions–worked to create the React components for their Figma prototype.
iSeaTree–a citizen science game-play app that lets users ID trees while contributing to the database–worked on selecting new data for their new CO2 and Hydrologics calculator feature. Volunteers reviewed over 300 tree images and tested the new version of the iSeaTree platform.
Nonprofit Exchange Hub–an app that helps nonprofits locate and exchange info and goods–made progress in the development of their prototype while refining wireframes for the next batch of development.
Orcasound–a website where citizen scientists can study audio recordings of Orcas and help to conserve them–developed a prototype app replicating the website experience.
Project Lockdown–an interactive platform providing an overview of the state of human and digital rights around the world–continued development on the MAP and API 2.0 versions, parsing the database and building new API models.
ShelterApp–a program that connects outreach workers to homeless encampments and offers real-time supply inventory tracking–worked on designing and developing a new Outreach Tracker app.
Turn the Bus–an app providing educational resources to high school students in underdeveloped parts of India–designed prototypes of a flashcard-style learning workflow, building a backend question database and moving the project closer to their Android launch.

What did our participants think?

Volunteering at a DemocracyLab event offers an opportunity to contribute to areas of interest and provides great career-building experiences. All participants found a project with a mission that they're passionate about; both volunteers and project leaders enjoyed the event and found it rewarding. Volunteers had the opportunity to learn by doing while contributing to tech-for-good projects impacting real issues they care about. Project leads saw tangible development and moved their technologies forward towards longer-term goals.

Survey results:

85% strongly agreed the experience was excellent
70% strongly agreed the quality of projects was excellent
70% strongly agreed they were satisfied with their team’s output at the end
85% strongly agreed they intend to continue working on the project
70% said they gained new experience that will benefit them

Learn more about getting involved

For more information about how you can participate in our next tech-for-good event please visit our events page.

To explore our list of projects with ongoing volunteer opportunities check out our website.

DemocracyLab is a volunteer-based 501(c)3 non-profit organization headquartered in Seattle, WA.

Optimizing Municipal Infrastructure with Smart City Planning

DemocracyLab — Wed, 07 Jul 2021 17:49:53 GMT

There’s a lot of buzz in the technology world about Smart City Planning. A Smart City collects and analyzes the data it produces to make improvements in areas like accessibility, sustainability, and walkability. During the pandemic countries like Singapore, Korea, and Taiwan used Smart City techniques to keep track of and contain Covid-19 infections.

The Smart City market is forecast to become a trillion-dollar industry, and many private companies are jumping in to create technologies to make cities more livable for residents. However, ethical concerns have arisen about private corporate control of cities' public data. Luckily people like Dr. Xuesong Zhou are working on open source projects, creating transparent alternatives that give the power to make data-driven decisions back into the hands of the community. Given that millions of people are moving to cities every week, efficiency has become an essential priority.

The Vision of a Communal Tool

Dr. Xuesong Zhou is an Associate Professor of Civil Engineering at Arizona State University specializing in dynamic traffic assignment, traffic estimation and prediction, and large scale routing and rail scheduling. He’s also passionate about the democratization of urban planning. He saw the potential of OpenStreetMap, an open source alternative to Google Maps, but knew it needed better functionality if it were to compete as a city planning tool. The Smart City Planning project was born.

The data packages built by Dr. Zhou and his partners extract the OpenStreetMaps data and add layers to make it more usable. To date, it’s been downloaded over 16,000 times from Github and is being used by universities and departments of transportation across the nation, including by Cambridge Systematics Inc. in the modeling system developed for Northern Virginia Transportation Authority’s TransAction Program.

Photo Credit: Smart City Planning

The Bridge Between Data and Community

For Dr. Zhou, the various problems affecting cities (traffic, pollution, accessibility) are related and solvable with data. By bringing together academics, community leaders, and non-profit organizations, he hopes to develop an open data source that will serve as the framework for running simulations and informing data-driven infrastructure decisions. Dr. Zhou has also been an avid participant in DemocracyLab’s hackathons, where volunteers updated the data package versions with additional data layers, such as traffic light timing.

That vision is already beginning to manifest. Through a program sponsored by Arizona State University, high school students have been able to contribute to Dr. Zhou’s project. Earlier this year, Xenia Zhao worked with Dr. Zhou to research food bank optimization in her local city and her paper won first place in the Arizona Science and Engineering Fair. The high school student currently working with Dr. Zhou, Rachel Dai, is using the Smart City data package combined with US census data to understand how to optimize accessibility for vulnerable and elderly populations

The Future of Smart City Planning

The long-term goal of the project is to build digital “twin cities” that mirror the physical space and infrastructure of a metropolitan area. Users will be able to easily run real-time simulations that model a city's utilities, transportation, physical infrastructure, civic services, and more. By building digital twins that are accessible and easy to use, the project enables community-supported city planning. Dr. Zhou hopes to continue partnering with high school and college students who are passionate about social change and who want to work on creative ways to use technology to improve their cities.

How to get Involved

The Smart City Planning project is currently looking for anyone familiar with OpenStreetMaps software or knowledge of Python, especially high school or college students interested in sharpening their skills with hands-on experience. Check out the project page on DemocracyLab and sign up to get started!

Supporting Diversity in the Health Sciences

DemocracyLab — Wed, 07 Jul 2021 17:40:21 GMT

Photo Credit: Northwest Health Career Path

Diversity in Health Sciences

There’s a lack of diversity in the American healthcare field—the rates of employment in healthcare don’t reflect current US demographics. Students of color and the LGBTQ community remain underrepresented in many healthcare jobs, and the number of medical students from rural areas is declining. Given that the White population in the US is projected to fall 10% in the upcoming decades, the gap is likely to widen if nothing is done.

Northwest Health Career Path (NWHCP) believes a healthcare workforce should reflect its community and works to support diversity in healthcare. NWHCP’s mission is to make it easier for underrepresented people to access the resources they need to successfully launch a career in healthcare. NWHCP has created one place to find everything from summer camps and after-school programs, to workshops and internships, as well as college access programs, and more. The website serves students from middle schoolers to postgraduates.

With NWHCP’s free interactive map, users can find nearby resources fitting their education level and career focus, as well as financial and transportation needs. Longer term, NWHCP hopes their initiative will facilitate collaboration between organizations to create a more complete ecosystem of support for these students as they navigate through the sometimes complex landscape of higher education.

NWHCP is a collaborative effort between the University of Washington Medical School Service Learning Program and several regional partners. (Check out the Support section of their website for more information about all of the organizations making this project possible).

How the Tool Works

NWHCP started by collecting data from all of the healthcare outreach programs across the region of UW School of Medicine’s WWAMI partnership (Washington, Wyoming, Alaska, Montana and Idaho), totalling over four dozen programs and organizations. The data is visualized on an interactive map, making it easy for students to find the resources closest to them. Each resource has a details page with a description and contact information.

How to get involved

Up to this point NWHCP has focused on getting their interactive map up and running. Next the project is looking to develop their website, and is partnering with DemocracyLab to connect with the following skilled volunteers:

Front-end developers
Back-end developers

Learn more about the project and discover how you can get involved by visiting the Northwest Health Career Path project profile on DemocracyLab.

DemocracyLab: Globally Inclusive

DemocracyLab — Tue, 15 Jun 2021 23:25:00 GMT

DemocracyLab has paved the way for thousands of volunteering opportunities through our projects, providing diverse types of roles and responsibilities. Once signed up, a volunteer from any part of the world can select the role that suits them best.

An inclusive culture is an indispensable component of recruiting as it leads to improved creativity, efficiency, and innovation. Diversity in the workforce unleashes immense potential for dynamic ideas, facilitating an organizational culture where each person feels free and comfortable expressing their thoughts. DemocracyLab encourages volunteers from around the world to get engaged in public interest technology projects. Such inclusivity and diversity are a core strength of the organization, which believes that diverse teams working together with goodwill and respect can accomplish remarkable things.

Through virtual tech-for-good initiatives, the volunteering opportunities at DemocracyLab open doors for many who want to make a mark by helping out their communities. With a vast selection of projects to choose from, aimed at supporting social, economic, environmental, and civic causes, remote volunteers have the chance to develop their skills while making a difference in the world.

Here are some of the inclusive behaviors that have helped DemocracyLab to encompass a worldwide presence:

Providing volunteer opportunities to people from diverse backgrounds and skills
Practicing inclusion strategies in recruitment, teamwork, and succession planning
Encouraging teams with a variety of skill sets by giving them opportunities to showcase their talent
Creating a sense of belonging

Ryan Lechner, content writer:

I am a writer who has contributed to DemocracyLab because of the opportunities available to learn and grow and its inclusive culture. I have been a part of this organization for around two years and can say that I definitely feel like an important part of it. There are many other international volunteers who feel the same about the work they do at DemocracyLab. Here are testimonials from some of our active volunteers about how the organization has made a difference in their lives.

Hakki, a UX Designer from Turkey:

I wanted to improve my international experience as a UX designer and also do some networking by joining DemocracyLab . Being able to connect and collaborate with people from various backgrounds is refreshing and interesting. I really enjoy working with people with various skill sets. Time difference is a challenge but DemocracyLab's inclusive environment helps me freely express my opinions and find opportunities to hone my skills. It sounds strange but without Covid-19, I wouldn't be in DemocracyLab. Because of the pandemic, it seems like we're moving into a physically distant but digitally more connected world. This is a borderless world where everyone can connect, collaborate, and create value regardless of their actual location. And I believe that in the long run it will have unprecedented results for everyone.

Zoe Nguyen, a UX designer from Germany:

I joined Democracy Lab to improve my skill as a product designer and I was happy to contribute my skill for good causes to society through the platform. I chose specifically DemocracyLab as I was inspired by the mission and the values of the organization. It creates a win-win situation for non-profit projects that lack resources as well as skilled people who want to practice some tech skills or simply want to help. Not to mention, I am amazed at how the platform grows by the teamwork of a diverse group of people and I am proud to be a part of the team.

Anee-Marie Nnaedozie, a UX designer from UK:

I joined Democracy Lab because of their cause and what the organization represents. I enjoy being part of the organization because of the causes and projects we inadvertently help by doing what we do. I’ve had the pleasure of working with great designers that I’ve collaborated with and learnt from; and would love to continue to do so.

Aashish Sawhney, a product manager from India:

I joined DemocracyLab as I was inspired by its mission and the work they are doing for tech-for-good projects. It's been a great experience so far to volunteer and give back to the community and I encourage more people to come ahead and contribute.

In such unpredictable, virtual-first times, it is not a bad idea for civic organizations to expand their circle with international volunteerism. The team at DemocracyLab takes diversity and inclusion to heart, and it shows in the incredible work we all accomplish together.

How Are Social & Environmental Organizations Growing with Data?

DemocracyLab — Tue, 15 Jun 2021 23:13:00 GMT

by Rhia June

As data analysis has become a more mainstream practice over the years, it has spawned an industry unto itself. University programs and online learning institutions alike have prioritized this field of study to meet the growing demand for data professionals, and a Maryville University's data analytics careers page makes it clear that this has resulted in data making its way into businesses of all kinds.

Organizations driving social and environmental initiatives are no exception, having opened their doors to data analytics experts to improve internal efficiency and external practices alike. While the applications for data analytics in this space are limitless, the fundamental idea is quite simple, as described in an article in The Conversation on data science: extract insight from data — that is, gather measurable information and determine what it says about performance and potential.

To fully understand the benefits, it's helpful to break data analytics down into simple steps to see how the idea is transforming businesses and organizations.

1. Gather Data:

The first step in any data analytics operation is to gather the data itself, which can be done in a number of ways, depending on the type of organization, including combing through transaction histories, assessing finances, monitoring customer feedback and social media conversation, and/or maintaining numbers related to website performance. These are just a few examples of the means by which company and organization leaders can begin to gather the data that will serve as the foundation for an analytics operation.

Keep in mind also that this shouldn’t sound overwhelming. While it’s true that gathering data requires some effort, and the more data the better, a lot of this process is typically automated. With the right tools and software, even a small organization can collect a great deal of valuable information.

2. Draw Insights:

People tend to imagine data analytics as a thoroughly advanced process involving complex computation. The not-so-secret truth though is that once an organization gathers its data, it can begin discovering insight with relative ease, often by simply turning raw information and numbers into something that can be interpreted clearly and at a glance.

This process is referred to as data visualization, which means turning data into charts and graphs. Creating simple visual representations out of unclear sets of information helps in analyzing the data and strategizing plans.

3. Apply Learnings:

Once an organization has its data in an easy-to-digest format, leaders can begin making key improvements.

+
Identify where operations are breaking down and falling short. Last summer’s blog post on ‘Using Data Analytics for Good’ touched on this very idea, noting that surveys in test communities had helped an analytics team to “understand what isn’t working and what is needed.” It really is about as simple as that. Whether in startups, civic engagement, or even big business, one of the key purposes of data analytics is figuring out what’s going wrong and why.
+
+
Identify growth opportunities. Spotting opportunities is in fact leading many organization leaders today to hire analysts or enlist volunteers from what has become a robust talent pool of data experts. As an example, a trained analyst might observe data that indicates the success of a particular outreach effort at generating engagement, and from there develop ways to repeat or imitate that effort on a continual basis. Someone in this role drives data-driven decision-making efforts, taking advantage of insights to revamp strategy and seize opportunity.
+
+
Improve targeting and outreach strategies. Data analytics is also used to target customers (or partners, donors, voters, and so on). This article by Towards Data Science delved into how targeting and outreach works, essentially pointing out that organizations can use data to glean demographic information about consumers. Whether this information relates to socio-economic status, purchasing history, location, interests, or any other factors, it can help organizations of all kinds sharpen their outreach efforts towards targeted audiences in specific ways that are more likely to work than a more general approach.
+

Any organization leader — of a non-profit, a volunteer group, an app-based company combating climate change, or anything similar — can appreciate the potential benefit of this sort of data collection. In the simplest of terms, more insight into operations (both internally and with regard to customers or communities served) lays the foundation for improved performance. All of these efforts are transforming organizations of all kinds, and will continue to do so for the foreseeable future.

WGU Helps Community College Students Hack for the Future

DemocracyLab — Mon, 14 Jun 2021 23:07:00 GMT

What was the event?

Western Governors University (WGU) understands that tech is the future. On May 15, 2021, WGU partnered with five community colleges and DemocracyLab to sponsor an online Hack for the Future event for tech students. The students had the opportunity to practice and learn new tech skills while improving tech-for-good initiatives ranging from civic infrastructure and social services to environmental research and conservation. Everyone who participated agreed the event was a rewarding educational experience.

How did the hackathon empower students?

WGU is reshaping education with its online performance-based approach. Their dynamic education model prepares students for the evolving demands of tech industries. A partnership with DemocracyLab was natural.

The virtual hackathon offered the perfect space for students to apply classroom learning to real world projects. The students worked in a collaborative environment with a narrow scope, and focused on making incremental progress toward the projects' long term goals. The students got hands-on experience contributing to the projects' code, building their confidence by applying their knowledge toward concrete deliverables.

Many students at the WGU Hack for the Future event also had the opportunity to work with new software or technologies for the first time. Furthermore, they made connections with the project leaders and other students that will last beyond the event.

What did the students think about the event?

The event opened pathways for a diverse group of community college students to learn by doing while contributing to important public interest technology projects. Following the event, students and faculty were surveyed, and two key themes surfaced:

Learn by doing: All students and faculty agreed the event was a valuable opportunity to apply learnings to real-world projects and to learn new technologies and skills.
Sense of worth: All the students surveyed agreed the event increased their belief they can make a difference in the world, and would participate in a similar event in the future.

As one student put it, “for this project, I realized, in that environment, I was the resident expert. And it felt great to be able to contribute!”

Students presenting the work they did during the Hack for the Future event.

What projects did they work on?

The students split into teams where they worked together with classmates and project managers to tackle a specific problem. Some of the skills the students used included bug testing and troubleshooting, building or testing open source code, security testing, and data analysis. The projects they worked on addressed a wide range of social and environmental causes:

A/B Street: a game simulation of pedestrians, bicycles, cars, and buses using real city maps where players edit lanes and intersections to improve traffic flow.
Every Voice Engaged: an online platform helping citizens, governments, and nonprofit organizations solve problems that benefit from civic engagement.
iSeaTree: a game-play app that lets kids learn, identify, and record trees in the USA and Canada.
Orcasound - Listen for Whales: an open source web -app that lets users listen for whales and act to conserve them.
ShelterApp: an all-volunteer, nonprofit organization helping homeless and low-income families connect to social services.
The Butterfly Project: a project analyzing public data and creating technology to mitigate the cumulative effects of systemic racism

By empowering students to become agents of change, WGU and DemocracyLab help students from underrepresented communities gain access to the booming tech economy and to challenge ineffective power structures.

Where can I learn more?

Solutions to many social and environmental problems will be driven by affected communities and empowered by tech. Shifting to online spaces allows for inclusive, global collaboration with expanding opportunities for people to contribute their talents to make a difference. Check out the resources below to learn how you can get involved.

Western Governors University is a nonprofit, regionally accredited online college offering over 60 degrees in Business, IT, Healthcare, and Education. Unlike traditional universities, students build on their previous knowledge and experience, and are assessed based on their competence in their chosen field. For more information, visit their website.

To get more involved in DemocracyLab’s projects, check out our upcoming events. Or if you’d like to contribute long term, check out our projects listings. Signing up for a project is easy and free.

If you’ve got a tech-for-good project in need of volunteers, simply create an account with us, upload your project, and start recruiting!

2nd Virtual St. Hack-trick’s Day Accelerates Civic Innovation

DemocracyLab — Wed, 14 Apr 2021 16:50:40 GMT

One Year Later, It's Clear COVID Can't Stop Tech-For-Good

March 3rd 2020 was our first day of Covid at DemocracyLab. Last year’s St. Hack-trick’s Day event was scheduled for the 14th, but with Seattle being the early epicenter of the outbreak we had a decision to make. We cancelled our reservation to use Code Fellows’ space and scrambled to find a way to make it all work online. You’re welcome to read our after-action report to hear how it went.

Now a full year into virtual hackathons, we can see that the pandemic accelerated us in the direction we wanted to go, but weren’t quite brave enough to leap toward - events with no geographic bounds. We’re excited to now see volunteers and projects from all over the world participating in our events. With vaccination becoming a reality, we’re looking forward to meeting the challenge of resuming in person events while maintaining the virtual element.

Innovating Toward Citizen-Created Government Technology

Though our geographic impact is expanding, our roots are still in the Pacific Northwest and we were thrilled to have three projects associated with state and local governments represented at 2021’s version of St. Hack-trick’s Day.

CiviForm is a responsive web application that will be accessible to users and their Trusted Intermediaries to help streamline applications to civic discount programs. A full team of Google.org fellows are supporting the City of Seattle to build this application as Seattle provides important resources to low income residents including utility discounts and child care assistance. However, the single portal that is currently used to apply and process for these resources is very inefficient. CiviForm hopes to improve the application process by allowing residents and their Trusted Intermediaries to apply to multiple City of Seattle assistance programs without needing to re-qualify for each. Also it aims to improve the efficiency in the application review and approval process for program administrators who work for the City of Seattle.

Portland Open Data STREAMS uses an evaluation framework for open datasets published by the city of Portland to create a data analysis tool which provides quality assurance for datasets made available between bureaus and to the public. This project is needed as the City of Portland currently has a large amount of public data across its various agencies but accessibility to that data is low for many interested partners and the general public.

SimpleForm is going to be a centralized web application with form entry proposals for CERB clients. This is needed as CERB provides funding to local governments and federally-recognized tribes for public infrastructure which supports private business growth and expansion. However, the current system is inefficient in terms of archiving and receiving forms. This is because CERB has to send emails to its clients with forms to be filled out and sent back. Then, someone at CERB has to manually record this in their personal database.

123 Volunteers + 16 Projects = 1 Impactful Event

In addition, there were tech-for-good projects, including consistent contributors like Orcasound and CDP and newcomers like Civic Tech Index and Project Lockdown.

The full list of participating projects included:

A/B Street: A game simulating pedestrians, bikes, cars, and buses in Seattle. Players edit lanes and intersections to improve traffic flow.

Accessibility Ratings: A crowd-sourced application for accessibility ratings will help users and organizations adopt and use more accessible applications.

Civic Tech Index: A global index of every useful civic tech open source software project.

CiviForm: Collaboration between the City of Seattle and Google.org to create a universal application tool to streamline benefit applications.

Council Data Project: An entirely free and open application to make it easier to follow your local municipality's council action.

DemocraticLi: Platform for location-based community organizing: Location-specific voting Information and discussion.

EveryVoice Engaged: A platform that helps citizens, governments, and nonprofit organizations solve problems that benefit from effective civic engagement.

iSeaTree: A game-play app that helps kids learn, identify, and record trees in King Co., WA. Works with USFS iTree CO2 calculator.

Nonprofit Exchange Hub: A hub for the exchange of goods, information and connections for nonprofits of all sizes.

OpenSidewalks: OpenSidewalks collects open (pedestrian + accessibility) data for data-driven social justice and equity analytics.

Orcasound: An open source web-app that lets users listen for whales and act to conserve them.

Portland Open Data STREAMS: A platform to support data preparation and data submission to publish City of Portland open datasets.

Project Lockdown: Project Lockdown is a civic tech, interactive platform providing an overview of the state of Human and Digital Rights around the globe.

SimpleForm: UW iSchool Capstone: assisting Community Economic Revitalization Board with a form entry system for better handling of CERB-related projects.

Smart city planning: OSM2GMNS aims to enable a rapid creation of routable multimodal networks and demand in General Modeling Network Specification.

The Butterfly Project: If we are to become a more equitable and just nation, then we must create technology to mitigate the cumulative effects of systemic racism.

Abundant Positive Outcomes

The diverse group of designers, developers, product managers, researchers and citizens who contributed made worthwhile connections, and helped projects make progress toward their long-term goals. This positive sentiment was echoed by a post-event survey of participants, which found that:

94% of respondents to our post-event survey agreed that they were satisfied with the work completed by their team.
100% of respondents said that they continue to work on their hackathon project.
94% of respondents agreed that their overall experience at St. Hack-trick's Day was excellent.

We’d like to sincerely thank all 123 project leaders, volunteers and sponsors who made this event a success!

Undebates: Making Democracy More Accessible

DemocracyLab — Wed, 14 Apr 2021 15:57:21 GMT

On Demand Debate Content Demystifies Candidates

Have you ever gone to vote in an election and realized that, for certain offices, you have no idea who the candidates are? You probably know the presidential candidates, and maybe even the gubernatorial or mayoral candidates; these candidates often get lots of free media coverage, as well as donations they can use on advertising. But what about other offices, like school board director or sheriff? These candidates don’t get nearly as much media coverage or money for advertising, so they often fly under the radar. If you’ve ever skipped over these offices on the ballot simply because you didn’t know who the candidates were, you’re not alone. For the candidates running for these lesser-known offices, it’s hard to be seen by voters. The media coverage isn’t there, and social media advertising is expensive and difficult to get right. And with over 500,000 elections across the country, plenty of candidates – and voters – are not able to participate in the democratic process to their full potential. So, how do we solve this problem? Enter: Undebates.

How Undebates Works

Undebates is a way for voters to quickly get to know candidates and for candidates to gain exposure to voters – for free! It can be deployed for elections all across the country.

Undebates worked with Ballotpedia, a portal which voters already use to learn about the candidates before they vote. Ballotpedia covers elections for all federal and state offices, plus the 100 largest U.S. cities. In partnership with Ballotpedia, Undebates creates a customized link for each candidate. The link leads to a video interview which guides the candidate through several questions, the responses to which are recorded. It also gives the candidates the opportunity to rerecord their answers and review everything before posting.

After a candidate’s answers are posted, they automatically become part of the Undebate for that office, and the link to that Undebate is then included in the info page for that candidate as well as the info page for that office if more than one candidate has recorded. (On Ballotpedia, these are called “Candidate Conversations.”)

When voters access the Undebates, they look like a regular debate. But Undebates move quicker, and voters can skip candidates or questions, as well as go back and review them if need be.

How DemocracyLab Helped Undebates

The idea for Undebates came about in August of 2019, and the MVP (minimum viable product – or prototype) was ready in time for the Nov. 5, 2019 elections. The first Undebate, which involved the candidates for the San Francisco District Attorney’s race, looked like this:

At the November 19 Hack To Give Thanks, hosted by DemocracyLab, the developers of Undebate worked with two designers, one of which joined the Undebates team, to improve the design.

Now the user interface looks like this:

In March, Undebates teamed up with DemocracyLab again for the 2020 St. Hack-trick's Day with the goal of improving its email campaign to invite candidates to do interviews. With the help of two volunteers, the team was able to implement email automation through Klaviyo with an email campaign that looked like this:

This effort resulted in videos from two of the mayoral candidates for Onalaska, Wisconsin.

Through 2020, Undebates worked with volunteers from DemocracyLab, HackforLA.org, and Taprootplus.org. In the 2020 elections, 91 candidates recorded Undebates for the Nov. 3 general elections, and 67 recorded for the primary elections before that.

But the Undebates team isn’t stopping there. Plans for the future include a self-service portal where election leaders for student council, homeowners’ associations, unions, and other democratically run organizations can create their own Undebates. Undebates also wants to support Americans with Disabilities in its user interface and by making transcriptions more accurate.

Through volunteer contributions to the open-source code base, Undebates hopes to make democracy work better. The DemocracyLab team is proud to have been able to assist in such a vital goal, and we look forward to making elections more accessible for all voters in the future. Learn how you can get involved by visiting the Undebates project profile on DemocracyLab.

iSeaTree Nurtures Community Scientists

DemocracyLab — Fri, 05 Mar 2021 17:29:38 GMT

The TreeMama's Dream - Identifying the Urban Canopy

Arielle Simmons-Steffen has a vision: engage a community of data-informed citizens in protecting trees, starting with the youth of North America. A self-described lifelong naturalist and former software engineer at Tableau, Arielle has worked for the Seattle government for 3.5 years, and is currently a Civil Engineer with Seattle Public Utilities. In 2020, she learned that a tree survey internship program run by the city had been cancelled due to COVID-19 concerns. That's when she realized that the software the youth program had been planning to use had no universal mobile interface. Thus, the idea for iSeaTree was born. Simmons-Steffen originally pitched the idea for the app to the Seattle government for funding, but when the coronavirus pandemic hit, there was understandably little room in Seattle Public Utilities' budget for anything beyond helping citizens keep the water on. Simmons-Steffen turned to DemocracyLab’s volunteer network to help get the app off the ground.

Simmons-Steffen credits DemocracyLab as being a “critical infrastructure piece” in the development of the app. The iSeaTree team participated in two of DemocracyLab's bi-monthly public hackathons and was able to add three long-term junior developers to their crew. The team then participated in our Hack for Good event with Amazon in December. Through the Amazon hackathon, the junior developers were able to work with and get advice from seasoned Amazon employees, which Simmons-Steffen is grateful for. “DemocracyLab helped us build our core dream,” Simmons-Steffen said. “You came in at the perfect time.” iSeaTree is now in its second version and is racing toward version 3 in time for tree-measuring season in the spring.

A young conservationist in the field

How iSeaTree Works

iSeaTree is available for free on iOS and Android and is designed for all ages and skill levels. Users simply record information about the trees they find, such as species and diameter at breast height (DBH), and then are rewarded with special badges and fun facts about the tree species. For those who are new to tree surveying, the iSeaTree online forum can help with species identification. Simmons-Steffen connected the mobile interface to the U.S. Forest Service iTree API, allowing users to access tree benefit calculations while in-field, as well as save the data they collect.

Simmons-Steffen’s primary goal with iSeaTree is to develop a smart, young usership that’s interested in learning manual tree identification, and not just relying on machine learning algorithms (which are often wrong and still require correction). What better way to engage a young population and make remote learning fun than by making it a game? Simmons-Steffen wants the data collectors using iSeaTree to be just as rewarded as the people using the data, and the iSeaTree team is currently working on expanding gameplay for future versions of the app. Simmons-Steffen particularly hopes to increase usership, especially among students and educators. Now that the app is functional in the United States and Canada, she hopes to see usership expand geographically as well.

Screenshots from the iSeaTree app

The Future of iSeaTree

Beyond expanding usership of the app, Simmons-Steffen hopes to partner with an established conservation agency to further the growth and adoption of the tool. Her philosophy is that an engaged population is more vital than the data collection itself, and she hopes that other companies and projects will adopt this mindset for their products.

Interested in helping with iSeaTree’s product, business, and community development? You can visit Simmon-Steffen’s Tree Mama blog here or iSeaTree’s profile on DemocracyLab and help build a dedicated community of tree surveyors.

Story by Kailey Miller

How Amazon Hacked for Good

DemocracyLab — Wed, 03 Feb 2021 08:30:00 GMT

Few companies have impacted our world as significantly as Amazon. On December 10th and 11th, 2020 Amazon's Self Service Performance Ads (SSPA) team participated in a virtual tech-for-good hackathon where they devoted their time and talents to making a positive impact in communities near and far. 52 Amazonians from around the world worked in teams to drive significant progress for a cohort of innovative tech-for-good projects. It was a win for the projects, for the employees, and for Amazon.

90% of project leaders reported making significant progress at the event
92% of employees agreed the event was a rewarding volunteer experience
73% of employees said the event increased their enthusiasm for working at Amazon
83% of employees said DemocracyLab provided an excellent environment and structure for the event

“At Amazon, we are continually looking for ways to achieve more with less, and work hard to raise the bar for customer experience. DemocracyLab provided us with an impactful way to use our skills to help non-profits achieve more, and contribute to improving the experience of their customers. It was time well spent – fun, impactful, and important.”

– Colleen Aubrey, Senior Vice President - Advertising Products and Tech, Amazon

DemocracyLab’s Mark Frischmuth outlines the event agenda for a group of project leaders and Amazonians.

How "Hackathons" Empower Employees

One of the first questions we often receive is “exactly what is a hackathon?” so we actually wrote an article about this! In short, it is when a group of volunteers band together under one roof - in this case, a virtual workspace - and work collectively on design and technical projects. DemocracyLab’s hackathons are different from most. Projects and participants are encouraged to work collaboratively rather than competitively, and focus exclusively on technology projects that advance the public good.

Countless studies have found that employees don’t just want to work, they want to engage in activities that make a positive impact on the world. A recent PwC survey found that 65% of people across various countries “want to work for an organization with a strong social conscience. Companies that promote active corporate volunteer programs see higher employee morale and a healthier, more energetic culture.” Additionally, a LinkedIn study noted that “while establishing employee volunteer programs is an obvious way for companies to give back to their communities and build their brands, there’s an additional benefit: volunteer programs are a powerful engagement and retention tool. In fact, when employees feel connected to their company’s giving and volunteering efforts, turnover drops by an average of 57%.” Furthermore, the study found that 79% of candidates “would prefer to work for a socially responsible company.”

The benefits of organized volunteerism don’t stop at employee morale and retention, though. They go all the way to the bottom line. A recent Deloitte case study found that 91% of respondents (executives and employees) who said their company had a strong sense of purpose also said their company had a history of strong financial performance.

Amazon has repeatedly shown itself to be a leader in finding ways to empower its employees to change the world, because they realize the importance of doing so. One of those ways was partnering with DemocracyLab on an internal Hackathon with their SSPA team.

Which projects did the Amazon SSPA team work on?

For the Amazon SSPA Hack for Good event, members of the DemocracyLab team vetted projects that were interested in participating, then selected projects that were a good match for the interests and skills of the Amazon engineers, managers, designers and user experience professionals that were participating. Members of the Amazon SSPA team were then matched with projects based on their interest and the scope of work each project articulated.

A/B Street: A game simulating pedestrians, bikes, cars, and buses in Seattle. Players edit lanes and intersections to improve traffic flow.
Banana App: A location-based food donation app system for imperfect food.
Every Voice Engaged: A platform that helps citizens, governments, and nonprofit organizations solve problems that benefit from effective civic engagement.
Fire Refuge: Fire response project to help evacuees find temporary overnight lodging.
iSeaTree: A game-play app that helps kids learn, identify, and record trees in King Co., WA. Works with USFS iTree CO2 calculator.
OpenSidewalks: Collects open data relating to pedestrian accessibility for data-driven social justice and equity analytics.
Orcasound: An open source web-app that lets users listen for whales and act to conserve them.
ShelterApp/Strappd: A nonprofit organization helping homeless and low-income families connect to services.
Turn The Bus: An Open edX-based app for video-based learning for students in India. The app provides tools to motivate students to learn, such as a leaderboard, badges, points and other online incentives.

Bratati Ghosh of Turn the Bus introduces an ecomm page mockup the volunteers created over the span of two days. Ghosh hopes to build a donation arm of the edtech website to sell crafts created by students’ parents.

How did it go?

Very well! Thanks to well-prepared project leaders and passionate and hardworking Amazon employees, the event resulted in immediate and tangible progress. Here are some interesting additional stats from the participants, gathered by survey after the hackathon:

100% of project leaders said that the quality of work they accomplished was significantly greater than at previous hackathons in which they had participated
60% of project leaders said that the quantity of work they accomplished was significantly greater than at previous hackathons
92% of employees said the lineup of participating projects was excellent
92% of employee agreed that project leaders were well prepared and easy to work with
83% of employees said they made meaningful contributions to the project they worked on
75% of employees stated they would love to participate in a hack-for-good event again
75% of employees said the event increased their belief they could make a positive impact in the world
50% of the Amazon employees who worked on projects during the hackathon said they intend to continue to work on the projects

Hack with Us!

The DemocracyLab platform features over 130 tech-for-good projects in various stages of development. In addition to corporate events, DemocracyLab convenes public tech-for good hackathons every other month. Over the past two years, DemocracyLab has held 15 public hackathons, engaging more than 1,500 volunteers and 200 tech-for-good projects. Hackathons have become an important program in our mission to empower people who use technology to advance the public good. Through our platform and programs we harness the powers of civic engagement and collective intelligence to create projects that have real, immediate and lasting impact. The DemocracyLab platform makes it easy to discover projects, match skill sets, and create teams.

Browse the DemocracyLab project catalogue here—signing up for a project is quick and easy!

Harness the power of Corporate Hackathons for your business!

Corporate hackathons are a perfect way to strengthen employee satisfaction by giving employees an opportunity to leverage their unique skills while improving their communities and the world. It’s a win-win for everybody! If you are interested in learning more about the value of Corporate Hackathons, visit this page or contact us.

The Mayo App - Creating Purposeful, Supportive, In-Person relationships beyond Screens

DemocracyLab — Tue, 17 Nov 2020 10:00:00 GMT

In an era filled with digital relationships and social isolation, physical in-person communications with neighbors and local strangers seem to be a thing of the past. Ironically, the people that can help the quickest when we are in any trouble are the people physically closest to us. Technology can get us personally connected to many strangers internationally, but the team at Mayo believes that one thing that technology should be helping us do more of is to connect with strangers within our close physical presence through helping each other.

Mayo is an app built and marketed by 20 people and the Heroic Together Community. Headquartered in Seattle and founded in 2018 by Allen Chan, a designer that formerly worked at Microsoft and Amazon. Chan’s app went on to be selected for the Third WTIA Founder Cohort and praised in Millennial Moderator for its small area focus and unique model which in turn makes “the accountability of users...much higher, and the sense of interacting and being able to see real people heightens.” The Millennial Moderator also adds that what Mayo does for its users can help them develop communication skills necessary for interactions in college, at work, and in their industry.

How to get and give help with Mayo

Mayo’s mission is summed up as the following on their ProductHunt.com page: “Help your community. Feel Happier.” The app itself is very user-friendly; all someone needs to do is post a request or offer for help; this can be anything such as looking for a lost wallet, asking for help on a college assignment, borrowing a charger, ask directions, or even saying you just want to spend time with someone. People with a 2-block radius would reply within an hour and can work out the details via the texting feature within the app. This straightforward use removes social barriers and makes introductions easy. Additionally, there no need to sign-up for an account or set up a profile, and the app offers ‘Kudos points’ for future rewards. Similar to Uber, users of Mayo can also review the experience of those providing assistance.

Mayo during the Coronavirus Pandemic and Beyond

Chan encourages people in communities to invite them to use the app and to spread the word online and in-person wherever possible, in addition to sharing positive “Mayo Moments” of people using the app. As the app is very useful in high-population environments, Chan has considered integrating the platform in meet-up environments and universities. In the meantime, Chan and his team are continuously improving the app to increase genuine real-world connections for people.

Mayo is also taking steps to be of assistance during the Coronavirus Pandemic. They share articles on their blog on how to ask for help, and how to help others while also staying safe.

“...Some people are desperately hungry. Some people cannot afford medical treatment. Most people are living paycheck-to-paycheck. These are the 'overlooked' who are most affected by this global pandemic. Now is the time to stop, reflect, and offer help to those who need it the most.”

The team at Mayo developed an app to create a positive social impact on others; now is the time where people need those positive social impacts the most.

Interested in helping with Mayo’s product, business, and community development? You can contact them via this link and contribute to a world where it’s easy for people to ask and provide help.

Join us at our Hack To Give Thanks this upcoming November 21 to help projects just like Mayo!

One With The Keyboard

Fri, 04 Nov 2022 12:30:00 +0200

one with the keyboard

a necessary prerequisite

+ + https://leafbytes.com/assets/home-row-e938435b.webp + Home Row Position + +

+ "What I was proud of was that I used very few parts to build a computer that could actually + speak words on a screen and type words on a keyboard and run a programming language that could + play games. And I did all this myself." +
-Steve Wozniak +

+ When I decided to dive in deep, and make my best attempt at learning everything I could about + Computers, I had already been, what is commonly called, a "Power User" of computers (at least + when it came to the Windows Operating System), and so didn't have as tough a time as some would have + when I started my journey into learning the intricacies of computer programming. I have been a more + than adequate typist since the age of eight years old and had taken the time to learn more + than a few keyboard shortcuts. These skills would prove invaluable as they would compound + on the skills necessary for basic desktop operations, and eventually, programming. +

+ There is such a huge skill and knowledge divide between the average everyday user of today's + computers and today's programmers, and that divide is becoming ever wider with the emphasis placed on + tablets, phones, and touchscreens. In many ways, the ability to adequately type is becoming a + lost art form that only college graduates are capable of, let alone the amount of those who + know about basic keyboard shortcuts. These skills, while not uncommon, are also not as ubiquitous + as they should be, especially when considering the prevalence of computers in our lives. +

+ So where do you get started when learning to code? If you already can type adequately without + looking down at your hands, and are familiar with some basic keyboard shortcuts, then honestly + you are miles ahead than those who start this journey with absolutely no knowledge of keyboards, + let alone computers. But this blog is written by a beginner computer programmer. So come, let + the blind lead the blind..., and let's cover some basics. +

+ Learning to type adequately and get to the minimum threshold of being able to comfortably type + 40WPM will take about two months of dedicated practice by my estimation (if you are a complete + beginner), practicing at least 2 hours a day. Other than a desktop/laptop computer and a + screen, you will(obviously) need a keyboard to get started. You should purchase a keyboard that + is comfortable to you, but, ultimately, any modern keyboard will do for the purposes of + learning how to type as long as it has the standard QWERTY layout(if the upper left hand row of + your keyboard spells out QWERTY, you're good to go). +

+ The next resource you'll need is a typing tutorial/course. These can be paid or free, but I'd + recommend that you simply go to any of the multitude of free online resources. I personally like + typing test. +

+ At first you will find yourself typing slowly and looking down at the keyboard often. Do not + fret, and do your best to not get frustrated. Learning a new skill takes time and is rarely a + comfortable experience. Whenever you find yourself looking down at the keyboard, return your + fingers to the home row, place your index fingers on the j and f keys, feel the raised bars at + the bottom of the keys with your fingers to remind yourself of the tactile sensation of being + in the proper starting position, and continue to memorize where each key is. Eventually this + muscle memory will become engrained in you, and you will be more focused on what you are + trying to type, rather than the location of the keys. +

+ Once you feel you can write a short essay (2 pages long), at about 40wpm, you are ready to move + onto learning some basic keyboard shortcuts. This is markedly easier than learning to type, but + provides you with an ability to navigate applications (particularly the web browser and word + processor) much faster, and ultimately this makes doing research and editing documents much + easier, with the added benefit of making you more productive as your typing speed is no longer + a bottleneck. Here are some of the basic keyboard shortcuts that exist on the most common desktop + platforms: +

+ On the Browser: +

ctrl+tab (cycle forward one open tab)
ctrl+shift+tab (cycle backward one tab)

+ Standard Word Processors: +

ctrl+a (select all text)
ctrl+c (copy selected text to the clipboard)
ctrl+v (paste selected text from the clipboard)
ctrl+z (undo last action)
ctrl+right-arrow (navigate to the next word)
ctrl+left-arrow (navigate to the last word)
ctrl+shift+right-arrow (highlight and navigate to the next word)
ctrl+shift+left-arrow (highlight and navigate to the last word)

+ On the desktop: +

super(windows/command key)+tab (navigate through the open applications)
super(windows/command key)+shift+tab (navigate through the open applications reversed)

+ The skill of typing and knowing certain keyboard shortcuts are not necessary to + learning how to code, but having a basic proficiency in typing will make learning + to code much easier. An approachable way to think about the learning process when it comes to + typing is that it is very much akin to learning how to walk. Using the mouse can be thought of + as crawling, typing as walking, and utilizing slightly more advanced tools like keyboard + shortcuts and text editors like Vim as running. All this said, there is a point of + diminishing returns when it comes to the correlation between quality of code and typing speed. + At some point you are reading code and looking at code and thinking about code much more than + simply typing it. You do not need to have the typing speed of a courtroom stenographer to learn + how to code, but it is essential that your typing speed (or lack thereof) not hinder your train + of thought when you are writing code. The location of the keys must be + subconscious in your mind so that the only constraint is the speed of your thoughts, not the + speed of your fingers. +

+ I was once told that learning to code and program was like learning to wield a great lever. + Once I had learned the skills to a certain capability, I would be able to do a lot with a + single flip of a switch. I believe this to be absolutely true. This makes the importance of + learning to type ever the more important. Until communicating with computers is done in a more + intuitive and immediate fashion, keyboards will remain one of the most important tools in the + modern world, as they are, undoubtedly, the predominant way we interface with computers, and + will be for the foreseeable future. +

Command Line

Sat, 05 Nov 2022 12:30:00 +0200

command line

getting comfy in the shell

+ + https://leafbytes.com/assets/Bash_Logo_Colored-47300f4e.webp + Bash Logo + +

+ "By this point, the shell had become a real programming language, suitable for writing + pretty much anything that could reasonably be formulated as a sequence of commands. + It could often do this well enough that there was no need to write a C program." +
-Brian Kernighan +

+ Learning to Program can be an intimidating venture. Not only is there a plethora of programming languages, libraries, frameworks, text editors, and more, there is also the overwhelming amount of opinions amongst more experienced programmers one might consider when deciding between which of these tools is best. Navigating all of this is no easy task and it is something I still struggle with when examining a field of software development I have little knowledge of (pretty much all of it). +

+ How does one know which text editor to choose? Which programming language should I learn first? Should I use MacOS or Windows? (hint: neither). One of the more uncommon (although not necessarily rare) questions asked amongst beginner developers is, which shell should I use? +

+ Before I dive into this topic, I feel like it may be prudent to address the simple question of, what is a shell? +

+ If you have done any amount of beginner programming, systems administration, + database management, or other computer related work, it is likely you have already + interacted with The Terminal (Emulator)/The Command Line and its close companion, The Shell. + I shall very briefly address what each of these are, and why, together, they are the most + important tool for anyone interested in interacting with computers at any meaningful level. +

+ In Windows, it is known as The Command Prompt(or Powershell), in MacOS it is known as The + Terminal, and in Linux and BSD, it is also known simply as the Terminal. The history of the + terminal is extensive, and I would only be doing you a disservice to you were I to attempt to + explain it in depth to you. For the sake of brevity, I shall simply explain how The Command Line + and shell pertain to you, the prospective Computer Programmer, and why it is so important for all + programmers to become familiar with it. +

+ If you were to open up Command Prompt in Windows, or the Terminal in MacOS, you would be presented + with a black (or blue) colored screen, usually prefaced by a series of characters known as a prompt, + and a cursor indicating the program is awaiting you to type something (to provide input via the keyboard). + To the uninitiated, this is intimidating, especially after inputting something and getting a mean looking + error message. I must admit, whenever anyone who is unfamiliar with computers has seen me working in the + terminal, they have often thrown around terms like "hacker." I find this to be so funny, because it simply + shows how media has failed to portray technology in an accurate light (save for modern exceptions like Mr. + Robot and Halt and Catch Fire, and even those are not completely accurate). Although hackers do indeed use + the command line, so do nearly all computer related professions! +

+ When I first began programming, I started focusing mainly on the tool of the text editor. + It was only when I began to seriously dive into the world of Linux that I began to play + around in the terminal. Even in my early days using Linux, however, I rarely interacted + with the Terminal. This is partially because for at least some time at the beginning, typing + in the shell felt laborious, more boring than programming (which it still is), and not worth + investigating beyond the few times I had to invoke a command, or start a program. And it's true + that the shell is just a tool that allows you to invoke commands that call other programs...but + that's why it will always be one of the best pieces of software in your tool belt. +

+ Just like everything in the world of programming, there is rarely just one option when it comes + to tools, and almost never just one way to do things. The same is true of shells, but just like + these other aspects of the programming world, it doesn't mean there haven't been some clear winners + in that space as well. +

+ Although I know I will get some push back on this from Windows Users, it simply is the case that + the UNIX-style shell won out. Specifically the BASH shell is the one most utilized by default in + nearly all Linux Distributions, and was also the primary shell utilized in MacOS until recently + (which currently utilizes the Z shell). Regardless of which shell you utilize, many of them have + similar commands that perform a plethora of operations via your operating system. It is due to this + common history with the BASH Shell, and the features that are available specifically within the BASH + shell scripting language, that it is worthwhile to dive deep into the subject of the shell and shell + scripting. The shell is your direct link to working on and with the operating system while utilizing + a high level language to do so. Bash, while not a full featured programming language, is an extremely + powerful scripting language which owes its (somewhat niche) popularity to the simple fact that GNU/Linux + is everywhere, and thusly, so is its default shell, bash. Indeed, it is due to the basic UNIX tool + set and the choice by the maintainers of GNU/Linux, BSD, and MacOS to adopt bash as their primary shell + of choice that it proliferated. +

+ At this point in the article I will digress on the subject of scripting, as a beginner is best suited + to learning a few basic commands, understanding some of bash's basic features, and how the use of the + terminal in programming is essential. Please note that while you can follow along in Windows, I will + not be covering Windows specific Powershell commands. Unfortunately Windows does not have BASH in its + software suite by default, and you must install The Windows SubSystem For Linux(WSL) or Cygwin to utilize + many of the same commands. +

+ Following the instructions below, note the effects. +

+ Navigate to your operating systems main menu +

(MacOS - Click the Launchpad Icon in the Dock, type Terminal and click Terminal)
(Linux/BSD - Look up which Terminal Emulator your distribution uses by default and open it)

+ You will be presented with a simple black, amber, or blue colored window with a prompt ending in either + % or $ (zsh uses %, bash uses $). There is also usually a rectangular cursor indicating the shell is + awaiting input from you. Typing just anything into the terminal and getting error messages gets old quick, + so let's explore our first command, ls: +

[~]$ ls

+ + https://leafbytes.com/assets/ss_ls_command-9c721e3b.webp + ls command results + +

+ Colloquially, each command will, in theory, be an abbreviation or shortened anagram for a word that will + remind you of the action often associated with the command. In this case ls stands simply for 'list' as in + 'list all my files and directories'. Here you can see that when I type in ls, I am presented with the contents + of what is known as my $HOME directory (we'll cover the $ later). Usually, the ls command has been + slightly modified so that colors appear on some of the results, indicating the type of file the result is, + or whether or not the result is a directory. +

+ If we then, for our second command, type 'ls' again, but this time follow it with a '-a', + we are presented with slightly different results: +

+ + https://leafbytes.com/assets/ss_ls_with_aflag-95942ec5.webp + ls command with -a results + + +

+ Here we see that we now have a lot more files and directories, many preceded with a '.', + which are our 'hidden' files and directories. There have been more than a few discussions + on the concept of hidden files, but needless to say, these files/directories are not visible + without the '-a' appended to our 'ls' command. I point this out as an introduction to the + concept of flags. '-a' is considered a flag of the 'ls' command that changes the results in + some fashion. In this case it stands for 'all', as in show 'list all of my files, yes even + the hidden ones.' +

+ Many commands, although not all commands, have flags which somehow change the behavior of the + program being called. +

+ I won't go into all the commands available in the shell as there are quite a lot, but I will + provide you with some standard commands to research and become familiar with, as they will become + invaluable later on: +

man
cd
mkdir/rmdir
touch/rm
cp
mv
echo/print/printf
cat/tac
less
head/less
read
find
sed
grep
awk
tr
df
top
ps
diff
dd
who

+ Now, all of these tools are great, but lack luster by comparison to your standard + Graphical User Interface (GUI) programs that most users interact with on their desktops. + The power of the shell isn't in that it can execute programs (although that is its main purpose), + it is its ability to chain together, redirect and manipulate the input and output of various programs + to yield a desired result...that's right, the bash shell also has a scripting language, which with the + use of various operators can chain together to manipulate textual inputs and outputs. + Take this simple example: +

[~]$ echo 'this text is inputted from the command line' from_the_command_line.txt

+ Here we simultaneously created a file called 'from_the_command_line.txt' and wrote the + text 'this text is inputted from the command line' into it. If we then inspect the + contents of this file using cat: +

[~]$ cat from_the_command_line.txt

+ + https://leafbytes.com/assets/ss_redirection_01-09a8114f.webp + cat command results + +

+ We'll see that indeed, the text 'this text is inputted from the command line' is + there. What is happening here? +

+ Well just like every problem in programming, we divide the problem up into sizeable + chunks, and explain each part to the best of our abilities (and hopefully our peers + very kindly correct us if we are wrong). +

+ Firstly, what does echo do? Well if you researched my list up above a bit, you'll know + that the first recommended command was the 'man' command. Why? Well it's the manual of + course! Any command you wish to know more about can be investigated by invoking 'man' + before the command you wish to know more about, if you type: +

[~]$ man echo

+ You'll see an extensive explanation of the 'echo' command and its various flags. A + relatively easy to understand program, echo simply displays any text following the + command back to you. So the input that was entered after the + 'echo' simply displays that input as the output. Thusly invoking: +

[~]$ echo 'this litle sheep went bah'

+ Displays: +

+ + https://leafbytes.com/assets/ss_echo_command-f890e6b2.webp + echo command results + +

+ But what about the '>' character in our initial example? Well this is what is + known as a redirection operator. This '>' will 'redirect' whatever is the output + of the first command(echo) as the input written to the following file or program. + Please note that this will overwrite any existing text already present in the file, + so don't experiment on any file with valuable information to you. If you do wish to + append to the file, without overwriting its existing text, simply double up on the + greater than symbol: +

[~]$ echo 'appended text' >> from_the_command_line.txt

+ + https://leafbytes.com/assets/ss_echo_append-6b223071.webp + echo with append command results + +

+ If you wish to research one of the most powerful operators in bash, research + the pipe ('|') operator! +

+ Hopefully at this point, you are starting to understand the role of the shell + and the command line a bit better. For this next example, I am going to demonstrate + how many programs have a command line interface with which one may interact, or at + least initialize, a program, be it a Graphical or Terminal based application. +

+ One of the first epiphanies for me regarding the terminal was when I realized I could + invoke pretty much any program in my operating system with simple commands. If I wished + to open my browser (librewolf), I simply had to type into the terminal its name: +

+ + https://leafbytes.com/assets/ss_librewolf-18d59d70.webp + librewolf browser being called from the command line + +

+ While this may not seem like a big deal to some, it was upon doing this that I realized + why the terminal was powerful, it allowed any program to be invoked as long as you knew + its name. Rather than clicking through a myriad of nested directories to find where I had + put an executable (like when using Windows File Explorer), I simply had to type it out and + it would open up on my screen. +

+ When I first installed Linux, the first piece of graphical software I had become familiar with + was the backup utility, Timeshift, which essentially takes a 'snapshot' of the current state of + your desktop and allows you to 'revert' back to that snapshot should you want/need to. +

+ Timeshift has both a Graphical User Interface(GUI) as well as a Command Line Interface(CLI). + I first learned to use Timeshift utilizing the GUI, but then I became curiouos about the CLI. + It wasn't long before I realized that most of the information available via the GUI interface + could be accessed via the CLI. I really only needed four commands, these were: +

timeshift --create
timeshift --delete
timeshift --list
timeshift --restore

+ And every once in a while, after an update on my system, I would invoke the 'timeshift create' + command to create a snapshot of my current system. Timeshift has occasionally saved me from big + mistakes I've made, and is a good piece of software. +

+ The reason I point this out is that even though timeshift isn't a part of the shell as a subject, + it is a practical example of when utilizing the text interface of Timeshift's CLI, via the shell, + became more useful to me than interfacing with the GUI version. This became even more the case when + I discovered the existence of shell aliases. +

+ Of course, I've provided you now with the knowledge of how to do your own research regarding any + command, so be sure to pull up: +

[~]$ man alias

+ And do more research, but I'll give you a general idea of why this is such a powerful tool + for anyone invested in computers to know. Essentially aliases are a series of commands invoked + by a custom command of your naming. +

+ If you invoke: +

[~]$ alias cdls="cd ~/Documents & ls"

+ You will navigate into your Documents directory and then list out its contents after + invoking your custom command, 'cdls' +

+ The '&' is another operator that waits for one command to complete before then invoking + the following command.The '&' operator can be chained as many times as desired and when + applied to an alias, essentially condenses a very long command into a shorter command (unless + you name your alias something very long I suppose). +

+ Now, the somewhat annoying thing is that these aliases, when created this way, are forgotten + as soon as you close the terminal or your shell session somehow ends. Aliases can be saved, + however, by placing them in your ~/.bashrc file. This configuration file is checked by the + shell prior to initiation, including alias assignments. Thusly, if you simply append: +

[~]$ alias cdls="cd ~/Documents & ls"

+ To the end of your ~/.bashrc file, you will have created a custom command that persists even + after you have closed your terminal and shut down your computer. +

+ This is the very essentials of scripting, the chaining of programs together to create different + results. An alias is essentially a one line script. +

+ While it wasn't one of my first aliases I ever wrote, I do have a single alias I will share with + you that I use often, it simply invokes timeshift three times, once to delete the old backup, once + to create a new backup, and once to list out all backups so I can confirm that the backup was made, + I called it tshift, this is the command in my .bashrc: +

[~]$ alias tshift="sudo timeshift --delete && sudo timeshift --create && sudo timeshift --list"

+ As you can see, tshift is far less characters to type, and it is a very handy, but very simple, + alias I have created for myself. (if you are on Linux and unfamiliar with the sudo command, please + see the man pages for sudo, if you are on MacOs, a simple query into your favorite search + engine I am sure will yield good results). +

+ Aliases is where I leave this short introduction to the Command Line and shell. It is at this + point that the subject of bash scripting should be covered, and that is an extensive topic of + its own. +

+ I do hope that you have now gained a better understanding and appreciation for the command line + and the shell. The shell provides a direct, unobfuscated, and elegantly simple interface through + which all computer users can accomplish more with less, and is one of the best tools you can + become familiar with. So get comfy in it! +

Why Use Linux

Sun, 06 Nov 2022 12:30:00 +0200

why use linux

learning to love the penguin

+ + https://leafbytes.com/assets/linux_mint-b5a145ad.webp + A Screenshot of the Linux Mint MATE Desktop + +

+ "You should be able to learn what your computer does. You should be able to make your software do what you want it to do (within the reasonable limits of its capabilities, of course). The key to attaining this power lies in understanding the fundamentals of what the software does and how it works.... You should never have to fight with a computer." +
-Brian Ward, How Linux Works; +

+ If you have ever taken even a small amount of time to research the world of computers in any kind of capacity, you will have at one point or another heard about the Linux Operating System. There is a certain mystique surrounding Linux and Linux users to the uninitiated, as it is considered by many to be more "technically challenging," or, perhaps more derisively, "too complicated and time consuming for the average computer user." These criticisms indeed have their validity, but anything worth having in life never came easy...and Linux is no exception. +

+ Much has been written on the subject of Linux, and this article, without a doubt, will be a simple reiteration of some of the arguments you may have heard or read when it comes to why you should use Linux, along with my own opinions on the subject. That said, should you be an aspiring web developer or computer programmer, please know that any of the popular operating systems are more than suitable to learning how to program. Be it Windows, MacOS, Linux, or BSD, nearly anyone can learn to program on any of these popular Operating Systems. I simply believe that Linux is the best Operating Systems to dive deep into learning about computers and to start programming. +

+ This article is meant as a general introduction to Linux, and in order to field the question of "What is Linux?" to the uninitiated, it's first good to step back and briefly field the more broad question of "What is an Operating System?". +

What is an Operating System?

+ I cannot under-emphasize the fact that I am generally new to the subject of computers, and thusly will probably butcher this explanation. I will, however, do my best to paraphrase and offer up a layman's definition of what an Operating System is. +

+ An Operating System is a piece of software that manages both the hardware and software on your computer. When you turn on your computer, the BIOS (Basic Input/Output System), a piece of firmware native to your motherboard, detects all hardware devices on your computer. Once all hardware devices have been registered, the BIOS then "looks" for a piece of software in memory where a program known as a "bootloader" is located. If found, it initializes this bootloader, which then, subsequently launches the "kernel". The kernel, at least for me, is a somewhat abstract concept that I oversimplify by describing it as being the "heart" of the Operating System. The kernel then launches the Operating System, which calls a series of programs, also known as processes or daemons (programs that run in the background), starting first with the "init" process. If all works well, you will be presented either with a Command Line Prompt, or a Desktop Environment. +

+ The most popular Operating System for desktop computers is Microsoft Windows, the second most popular being Apple's MacOs, and Linux comes in at third (BSD is fourth, but well below even 1% of overall desktop usage). It is important to note, however, that the majority of server computers around the world run Linux. (It is also of interest to note that both Google's Chrome OS and the Android OS are also based on Linux). The fact that Linux has come to be the predominant and preferred Operating System for server computers is one of the reasons I personally believe that it is a good Operating System to start on for learning about computer systems in general, although it is only one of many reasons which I will go into more detail later on in this article. +

+ The above explanation, to be clear, is a gross oversimplification, and I have no doubt that I am butchering it. For the sake of brevity, however, I will simply refer you to the following Wikipedia articles on the aforementioned topics should you desire a more in depth and succinct explanation on the topics: +

What is Linux?

+ Now that the subject of Operating Systems has been given the briefest of overviews, + let us now cover the subject of what exactly Linux is. As you probably have ascertained, + Windows, MacOs, and Linux are all Operating Systems that the majority of computer users are + familiar with from their interactions with their various devices, be it their Desktops, + Laptops, Phones, and/or Tablets. Each of these Operating Systems comes with a suite of + software, and as each Operating Systems has evolved over time, each has developed its own + sort of ecosystem, an opinionated approach as to how to address the variety of problems that + computer users and programmers encounter regularly. +

+ At this point, you might be asking, "If Linux is so great, why haven't I seen it around?" Well, the truth is you have. As I have pointed out before, Linux runs the majority of servers, and thusly, whenever you interact with a website or an application online, it is highly likely that the services you are interacting with are running on a Linux server (not to mention the predominance of Android devices which also all run Linux). The more apt question might be, "If Linux is so great, why haven't I seen it on Desktop/Laptops more often?" This is a more nuanced question, and rightly deserves its own article, but I will briefly address it here. +

+ Linux is not installed by default on most desktops/laptops. This is due to many factors, and there has been much discussion on why the "Linux Desktop" never took off. Indeed, "The Year of the Linux Desktop" is a meme within the Linux community, referring to an unspecified time in the future when The Linux Desktop will finally be realized as superior, and the majority of everyday computer users convert to Linux (something that is extremely unlikely to happen). +

+ Additionally, and perhaps more to the point, is the simple fact that Linux, being an extremely customizable system, has a "build it from the ground up," aspect to it, and the main interface for Linux will always be the command line, which requires (sometimes extensive) technical knowledge. Indeed, just to install Linux, you must download what is known as an ISO file, burn it to a USB stick (or other installation media), make some adjustments in your motherboard's BIOS, and then follow the installation instructions for that particular Linux distribution (at the end of this article, a brief introduction to this process is documented). +

+ Linux is a piece of Non-Proprietary Open Source Software under the GNU Public + License (GPL). This means that it is free to run, study, share, and modify however + you like (which is not the case when using Proprietary Operating Systems like Windows + and MacOs). The fact that Linux is Free and Open Source Software (commonly termed FOSS), + is one of the major reasons that Linux users at times can appear to be almost cultish or + emphatic about the superiority of The Linux Operating System. The philosophy around FOSS + can at times, in my experience, be seen as akin to a political or religious opinion, and + is held strongly by many within the Linux community. This, however, is not one of the + reasons I believe that it is worth your while to learn how to use The Linux Operating System, + but this phenomenon is something to take note of. This is because, should you choose to install + and use Linux, you will probably come across others within the Linux ecosystem who profess their + love for Linux with an almost emphatic fervor (I have been guilty of this myself). +

So Why Use Linux?

+ At this point, I've covered the most basic aspects of what an Operating System is as well as what Linux is, but I have yet to get to the heart of why you should use Linux if you desire to know more about computers and computer programming. As I pointed out towards the beginning of this article, you can indeed learn programming on Windows and MacOs, and many people do. So what makes Linux better? +

+ In one word: freedom. +

+ Many who are advocates for Free and Open Source Software (FOSS) are quick to point out that the "Free" in the title doesn't necessarily stand for "Free as in Free Beer" (i.e. Free as in No Charge), but rather that it stands for "Free as in Freedom." This has more to do with the fact that FOSS advocates are proponents for your freedom to modify the software however you see fit, which also includes your freedom to access the knowledge of how the software works. (As an interesting side note, there is a similar advocacy made by those among The Right To Repair Movement). +

+ Due to the fact that Linux is Free and Open Source Software, its entire source code is free to be run, studied, shared, and modified by its users however they see fit (at least on their personal machines). This is one of the major reasons to run Linux. While it takes (sometimes a great deal of) technical knowledge and a good deal of time to customize Linux, it is to your benefit to explore the Linux Operating System, as none of the source code is hidden from you, and you are free to modify the source code however you like (with both wonderful as well as disastrous effects depending on your intentions and skill level). +

+ This isn't to say that I'm modifying my Linux kernel on my personal machine (I'm not a Gentoo user), but rather that the knowledge is freely available for me to investigate, and this transparency is reason enough for me to use it. +

+ That said, I know for even the most technically minded user, this philosophical treatise on FOSS is not a good enough reason to abandon their Windows or MacOs desktop machine and embrace Linux. For most of us looking to become more computer savvy, we want software that simply works, and works well. Luckily for the sake of my argument, Linux is one of the best examples of this. +

+ Towards the end of this article I will be going over how to install Linux, but I want to briefly touch on what one gains from even just this basic tutorial. Let us think on what happens to the average computer user who wakes up one day to discover their Windows or MacOs machine has stopped working (they are presented with the dreaded blue screen or black screen of death). +

+ The average user may actually just throw away the machine without even thinking twice. They have money to burn and don't care about the environmental cost of chucking an electronic device. Others, who are a bit more budget conscious (or environmentally conscious), will take it into a computer repair service like The Geek Squad. If their Operating System is Windows, and the Operating System is irreparable (but the hardware is still functioning), the repair service technician will likely simply download the Windows ISO, burn it to a USB stick, and reinstall Windows (hopefully after backing up what they can and informing the customer). Of course, they will charge a modest fee for this, but the computer will be operating once again and the customer didn't have to do any research or work to do so, they just had to shell out a few bucks. Note that this is the same simple technique that is utilized to install Linux at the end of this article. By simply following the tutorial, you have gained a small piece of knowledge on how computers work and learned how to install an Operating System should your current one be somehow corrupted. +

+ Now, there is an obvious advantage to going to The Geek Squad and not + learning how to reinstall your Operating System. For one thing, you save time + (sometimes a lot of time). Time is a limited and invaluable resource and I will not + go into the obviously subjective question of how one should best spend their time. + Sufficient to say that when it comes to most computer users, they will not wish to spend + their time learning the ins and outs of their Operating Systems... but what about you? +

+ If you are a software developer, web developer, data scientist, and/or systems administrator, then it is in my strong opinion that it is worth your time to learn how to utilize The Linux Operating System, and yes, on your Desktop/Laptop! Building your home Linux computer from the ground up is an experience that will give you (at least a small) insight into the complicated inner workings of your hardware and software and how they interact with one another. Once installed, the base Linux system is customizable to a degree that dwarfs the customizability of Windows and MacOs. The act of customizing your Linux machine often involves configuring your chosen suite of software (sometimes from the source code itself), and slowly but surely, over repeatedly reviewing lines of code and configuration files, you will begin to understand on a slightly deeper level how it all works. +

+ This is a contentious opinion of mine, and there are legitimate arguments as to why one should simply utilize MacOs or Windows when first learning about computers in depth, and I'm not necessarily opposed to this argument. Rather, I am simply advocating that if you are going to learn about computers, then learning how to utilize the Linux Operating System is one of the best investments of your time you can make, as even in customizing your Linux machine, you will learn a decent amount about the inner workings of the software. +

+ This isn't to say that installing and using Linux has somehow given me a knowledge and insight into how the entirety of every piece of software that I utilize works (that would probably take a lifetime). I can honestly say, however, the amount of knowledge I've gained over the past two and a half years while ascertaining how to utilize The Linux Operating System has been vast, and I continue to learn more and more every day. The practice of reading and re-reading of documentation, the in depth discussions on various computer forums, the investigating of how to interface with a particular API, the interacting with the software itself, and the act of "getting your hands dirty" have all added up to the cultivation of a state of mind that is highly accustomed to learning any piece of software, and that mindset grew from my initial interest in understanding The Linux Operating System. +

Where To Get Started

+ Before I cover how to install Linux, I'd like to take a brief moment to cover Linux Distributions, colloquially known in the Linux Community as "distros". Earlier in this article, I mentioned multiple times that Linux is an extremely customizable Operating System. This extreme customizability has led to the development of a multitude of Linux "Distributions," which one can think of as different "flavors" of Linux. To the uninitiated this immediately can become overwhelming. As soon as you've decided to use Linux, if you were to turn to the online community and ask "Which distribution should I start with?," you might very well be bombarded with a myriad of answers. Indeed, if you look at the wikipedia article on the subject, you'll see how vast the world of Linux distributions is. +

+ The topic of Linux distributions is a subject on its own, and your choice of Linux distribution is a personal one (very much like the choice to use Linux itself). I will simply point to certain distributions that are generally recommended for beginners and then provide you with a brief tutorial on how to install Linux Mint (one of the more popular "beginner friendly" distros). Here is a brief list of distributions I recommend for beginners: +

+ If you are curious as to which Linux Distribution I run on my personal machine at home, it is Artix Linux, but I don't recommend it for beginners. +

+ Once you have chosen which Linux Distribution you'd like to try, be sure to read the documentation on its installation thoroughly. You will first need a working computer, an installation media (a usb stick has become standard for this), and a piece of software to burn the Linux ISO to the installation media (the easiest to use on Windows and MacOs is Balena Etcher, although Rufus on Windows is also a fine option). +

Installing Linux

+ This is just a brief overview of installing Linux Mint, while you can follow along here, I'd recommend reading Linux Mint's Installation Guide, as it gives much better and in depth instruction than I provide here. Please note that these installation instructions will remove your old Operating System and all files, software, and directories, so please back up your files prior to installation. While it is rare that your computer will be unable to run Linux, it does occassionally happen, so do some research on your particular hardware and its compatibility with Linux prior to making the attempt. +

+ The first step is probably the easiest, simply download the ISO: +

Download Linux Mint

+ The options you are presented with are Desktop Environments. Each one is slightly different, and each option deserves its own research, but the Cinnamon Edition is Linux Mint's flagship Desktop Environment and it is the one you will be downloading and installing if you choose to follow along with this tutorial. +

+ Once you have downloaded the ISO file. You're going to need to burn the file to a usb drive with enough storage capacity for the file (roughly 2.4GB). You'll also need a piece of specialized software to burn it. For this tutorial we will use Balena Etcher: +

+ Etcher is very easy to use, simply insert your usb stick into one of your computer's usb ports, open up the Etcher application, and follow its three step process. First click on "Select image", navigate to and select the ISO file within the presented File Manager. Then do the same for "Select Drive" (the usb device). Lastly simply hit "Flash!" and wait for the image to burn (this could take a while depending on your hardware). +

+ + https://leafbytes.com/assets/etcher-80153bf0.gif + A gif showing the basic usage of Balena Etcher + +

+ Once done your installation media should have the ISO image burned onto it. Inspecting the contents of the usb-drive should reflect this change. Keep the usb stick inserted into your computer and restart. +

+ As your computer restarts, you will usually see an introduction screen from your Motherboard manufacturer, when you see this introduction screen show up, quickly and repeatedly hit either the F2 or F11 key to enter the BIOS settings (if this doesn't work research your motherboard manufacturer, as it is usually one of these two keys). If installing on a Mac, keep your finger pressed on the Alt or Option key after hearing the boot sound. +

+ + https://leafbytes.com/assets/bios-94b8340a.webp + An example picture of an ASUS BIOS Menu + +

+ Once in the BIOS menu, you'll need to change the Boot Order so that your computer boots from the USB. To do this, locate and select the "Boot" Submenu. +

+ Once you have located the "Boot" submenu, locate the option labeled "Boot Priority". Under this option you should find a list of devices, one of which will start with USB, which is more than likely your installation media. Re-order the options so that the USB option is on top. +

A Word On Secure Boot

+ Some Linux Distriutions, in order to boot properly, will need you to disable the Secure Boot Setting in your BIOS. Please do your own research regarding this, but I have personally always disabled Secure Boot when installing Linux, and have determined it is a general prerequisite prior to installation. I briefly cover how to do so here, followed by a guide: +

+ Secure Boot is usually found under a section in your BIOS menu called "Security". From there select "Secure Boot" and the drop down menu should have an option labeled "Disable." Select this option. +

A Guide to Disabling Secure Boot with Pictures

+ Once you have disabled secure boot and re-ordered the bootable devices in your BIOS, go ahead and Save & Exit. The computer will once again reboot, and with any luck you will be presented with the Linux Bootloader: +

+ + https://leafbytes.com/assets/grub-731e44da.webp + A Picture of the GRUB Bootloader menu + +

+ Shortly after this you will be presented with Linux Mint's "live session" or "live environment." This is indeed the desktop environment, but you have yet to overwrite your previous Operating System, as the installation process is not yet complete. Do NOT remove the USB stick, as the live session is running from the USB directly. If you'd prefer to explore the Linux environment prior to installing it, please do so at this point if you are unsure about removing your previous Operating System and installing Linux. Most Linux Distributions come with a suite of software for you to play around and experiment with. +

+ + https://leafbytes.com/assets/live-environment-915bc82d.webp + A Picture of Linux Mint's Live Desktop Environment + +

+ If all is to your liking, and you're ready to install Linux and remove your previous Operating System, then continue with the next step: +

+ Select The Install Linux Mint option with the round CD icon on the Desktop. This will open up a menu that will walk you through the process, which I will not go into detail here, but generally involves setting up your internet connection, keyboard settings, timezone/locale, username, password and root password. Once you have filled out all form fields and hit "Install," the installation process will begin (this process might take a while). +

+ + https://leafbytes.com/assets/mint_install-c7c987a4.webp + A Picture of Linux Mint Install Menu + +

+ Once finished, Restart your computer and remove the USB installation media. +

+ If all went well, you will be presented with a login screen with your username already filled out. Enter your recently created password to log in. +

+ + https://leafbytes.com/assets/mint_login-0c98b545.webp + A Picture of The Linux Mint Login Screen + +

+ Once logged in, you should see the Linux Mint Cinnamon Desktop Environment, which is now running natively on your hardware. Congratulations! You have successfully installed Linux! +

On Updating

+ I will complete this article more or less here with one last bit of advice. Utilizing Linux generally involves interacting with the terminal (at least occassionally for casual users). Unlike Windows or MacOs, there are no automatic updates, you choose when you wish to update your software. Upon initial installation, it is always recommended to update your software. You can do this by navigating to the terminal application from your Launch Menu (located on the bottom left hand side of your desktop). +

+ Open up the Terminal and enter the following command to update your newly installed Linux Operating System: +

[~]$ sudo apt update && sudo apt upgrade

+ You will be asked for the sudo password, which is the same as the password you used to log in. Once you have entered your password, the apt package manager will query the Linux Mint Repositories, checking what software is out of date. It will then present you with a list of which pieces of software (packages) are out of date and ask you if you'd like to udpate them, enter "y" for yes, and wait for the installation process to complete. With the exception of certain pieces of major software, like the kernel, you do not need to reboot your system to use this newly updated software, it is available to use immediately. +

Conclusion

+ Thanks for taking the time to read this somewhat lengthy article. Hopefully I've instilled in you an interest and desire to use The Linux Operating System. I have only been using Linux on the desktop for about two and a half years now, but have learned so much in that brief period of time. I truly do believe that if you wish to get involved in any tech-related field that at least becoming familiar with Linux (if not using it on a daily basis), is essential. If you followed along and installed Linux Mint, welcome to the Linux Community! Should you have any questions/comments, please feel free to reach out to me! You can also field many questions over at The Linux Questions Forums.

Musings On Vim

Tue, 08 Nov 2022 12:30:00 +0200

musings on vim

one editor to rule them all

+ + https://leafbytes.com/assets/vim-370288c5.webp + The Vim Logo + +

+ "How do I quit this thing?" +
-Every Vim User At Some Point +

Introduction

+ Should you have chosen to start programming/coding, you will undoubtedly at some point encounter those with opinions on which software is best to utilize. While the amount of tools available to software developers is extremely vast, there are three (maybe four) essential tools necessary for a beginner software developer to get started (not mentioning the Operating System). These tools are a Web Browser (for research and web development), a Terminal Emulator (for interfacing with software, both your own and others), and a Text Editor (the fourth is a compiler, but I'll digress on that topic for now). +

+ Each of these essential tools have a wide variety of options, with a few clear winners that are extremely popular. In the Web Browser world, you have Google's Chrome, MacOs's Safari, and Mozilla's Firefox. The Terminal Emulator world is quite vast where the clear winners are less obvious, but two of the most popular options on Linux currently are the Alacritty and Kitty Terminal Emulators. Then we come to the world of Text Editors, which has a somewhat vast array of choices, with Microsoft's Visual Studio Code being the most popular by far. Other text editors include Github's Atom (likely soon to be deprecated at the time of this writing), Sublime Text, Microsoft Notepad++, and many others. When addressing the subject of text editors in online communities, you're likely to hear an overwhelming and increasing majority of users singing the praises of VS Code, perhaps a few chiming in about Sublime Text, the somewhat rare Emacs masochist waxing poetic about how Emacs isn't JUST a text editor, but a way of life. Then, of course, there's Vim. +

What Is A Text Editor?

+ To simplify greatly, a text editor is a piece of software for the purposes of generating and editing text. If you have ever played around in Microsoft Notepad, you'll be forgiven for thinking that it is so basic that it should only be used for quick notes and doesn't serve much other functionality. While this is somewhat true, you can indeed code in Notepad, as all you technically need to code is a piece of software that is capable of inputting and editing plain text. Text Editors do have some overlap in functionality with Word Processors, but the purpose of a text editor has far less to do with presentation (meaning there is little to no formatting for the intentions of printing, and rather is intended for the ease of use in coding and technical documentation). +

+ Modern text editors provide a basic suite of features by default. Some of the most obvious of these features are syntax highlighting (colorizing of certain keywords in the text depending on the programming language being edited), linting (helpful error messages that inform the user of possible errors in the code), and a vast suite of various tools for quickly navigating and searching throughout their project hierarchy. They are also highly customizable, with a wide variety of plugins available, as well as a configuration interface with which the user can further tailor the text editor to their specific needs. +

What Is Vim?

+ Obviously, given the track of the article thus far, you can easily ascertain that Vim is a text editor of some note. What makes Vim unique amongst text editors is its adoption of the original workflow from the Vi text editor, which heavily utilizes the feature of Modal Editing. Unlike other text editors, in which the user can immediately start inputting and editing text, Vi introduced the concept of "Modes," in which the input from the keyboard would not necessarily produce text on the screen, but would rather either navigate or manipulate the text in some way other than simple input. To be clear, Vi can still input text in the same fashion as other text editors, it's just that this mode (known as INSERT mode), is only one amongst many modes the user can choose to be in. On a modern text editor like VS Code, the navigation within the document from say, one word to the next, is accomplished using the 'ctrl + right arrow' keybinding. In Vim, while you can do the same in INSERT mode, if you are in what is known as NORMAL mode, you can traverse the same navigational pattern by simply typing 'w'. While this may seem trivial to the uninitiated, this modal editing is the major distinguishing feature of Vim with the general effect being that the user's fingers never leave far from home row, and thusly do not need to "reach" for keys on the perimeter of the keyboard as often, thusly increasing typing speed. Additionally, the "chaining" of these different modal keybindings while in NORMAL mode also increases productivity as one can more quickly navigate and edit a document than in a traditional text editor like VS Code. +

Vim's Drawbacks

+ Utilizing Vim's keybindings and modal editing is actually nearly ubiquitous across text editors now. There is a Vim Feature/Plugin for nearly every text editor, as well as a Vim setting in your shell (simply write "set -o vi" in your ~/.bashrc), and even your web browser can be modified with an extension to utilize Vim-style keyboard-centric navigation. With Vim's flagship keybindings having been ported to many other text editors, one might wonder why use Vim at all? This is a more compelling argument against using the original terminal based Vim text editor (especially if you also don't find the following argument below regarding the efficient of use of system resources compelling). Vim in the terminal requires more in-depth configuration than other text editors. While being more customizable than other text editors, the time spent in configuring any piece of software could easily be spent learning some programming language, or simply doing anything else for that matter. The scripting language, vim-script, is not portable to other programs, and thus the time you spend learning vim-script does not transfer a commonly used skill over to other projects (It is of note that Vim's successor, Neovim, is configured in the scripting language Lua, and thusly learning Lua through configuring Neovim may be a more worthwhile endeavor in this regard). In my previous article on Why You Should Use Linux, one of my main arguments behind learning to utilize The Linux Operating System was the fact that the act of customizing your Linux machine will help you learn more about computers in general. This same argument cannot be applied when regarding configuring Vim as a way to learn more about your computer, though in configuring Vim, you will learn more about your text editor. Ultimately, many of the features available in VSCode natively have to be implemented in Vim via plugins and extensive configuration. So what are the utilitarian reasons for utilizing Vim? +

Why Use Vim?

+ It almost goes without saying that the majority of new programmers/coders reach for Microsoft's VS Code as their first text editor. The predominance of VS Code is due to the simple fact that it is a great piece of software that works out of the box. Its ecosystem is vast, with a wide variety of plugins, color themes, and a helpful community. VS Code even has a Vim (and Neovim) plugin so one can utilize Vim-style modal editing keybindings. So, why, ultimately, would I suggest you utilize Vim(Neovim) over VS Code? There are a few good reasons. +

+Just like with choosing Linux, the choice of a text editor, like Vim, can become quickly political in nature due to the fact that VS Code is not entirely Open Source. While VS Code's source code is released under the permissive MIT license, the popular Microsoft release of this has proprietary software integrated into it. Additionally, VS Code utilizes telemetry by default, which goes against the FOSS world's advocacy for opt-in, and not opt-out, telemetry (The fork, VS-Codium, turns off this telemetry by default). +

+ On a less political note, Vim, and its popular descendant, Neovim, run natively within the Terminal and Terminal Emulator. This greatly reduces the amount of system resources utilized during operation when compared with VS Code, which utilizes the resource-intensive Electron as its base. This alone, is reason enough to utilize Vim. I have a relatively powerful machine with a modern CPU and GPU, but gravitate towards software that runs on as little resources as possible. Many argue that system resources are cheap today, and needing to squeeze every bit of memory out of your machine is a thing of the past, but this is a short-sighted argument. System resources are always disposable until you run out of them, and programmers in particular should never view system resources as being somehow magically infinite. This isn't to say that VS Code is a memory hog per say (that is relative), but the majority of what one can accomplish in VS Code, one can accomplish just as well, if not more efficiently, with Vim, and at a fraction of the system resource cost. The argument regarding system resources, however, is moot when compared to the next argument I will make in favor of using Vim as your text editor. +

+ Vim, with it's native modal editing ecosystem, incorporates a series of keybindings and shortcuts that greatly increase the speed with which the user can edit text. Much has been written on how to get started with Vim, and I will be providing you with a basic introduction to Vim towards the end of this article, but Vim is not an editor that is as intuitive to get started with as VS Code, and this is one of its major drawbacks. Although one could make the argument that learning Vim is similar in nature to learning the basic keybindings native in most Windows and MacOs applications, I would argue that Vim has a far more vast and rich set of keybindings, which, as mentioned above, can be chained together to achieve rapid results whether navigating or editing the document. +

Vim Basics

+ For the following short introduction to Vim, I will be utilizing the modern fork, Neovim, but for the purposes of this tutorial, you can use either version. +

+ How you install Vim will depend on what Operating System you have. If on Windows or MacOs, please take a look at these instructions, or you can install NeoVim. +

+ If you're using Linux, utilize your native package manager: +

+ Debian-based Linux distributions: +

[~]$ sudo apt install neovim

+ Arch-based Linux distributions: +

[~]$ sudo pacman -S neovim

+ The text editor will open with a brief introductory screen. + For those more accustomed to VS Code or a similar Graphical + User Interface(GUI) text editor, Vim can appear rather + sparse. At least the introduction screen provides us with + some information abut how to get started (:help vim ). I would encourage you to take a look at this help page, as Vim and Neovim documentation is quite good. I will now cover some of the basic keybindings and Vim "movements" in concluding this article, in order that you might get a sense of Vim's appeal. +

+ + https://leafbytes.com/assets/ss_nvim_01-c106224a.webp + The Introduction Screen to Neovim + +

+ When you first attempt to type anything into Vim, you may very well be immediately frustrated at the fact that no text appears on the screen! What the hell?! Isn't this supposed to be a text editor? Well, hold on, remember what I said earlier about it being a MODAL editor? Well if your cursor is in the default white block state, then Vim is currently in NORMAL mode, which one can think of as being a basic navigation and editing mode. In order to type something out, press 'i' in order to enter INSERT mode. +

+ + https://leafbytes.com/assets/ss_nvim_03-a61c0e8c.webp + Neovim in Insert Mode with Sample Text + +
+ + https://leafbytes.com/assets/ss_nvim_02-8421ef8b.webp + CloseUp of the INSERT mode prompt + +

+ Changing over to insert mode will change the shape of your cursor to the thin bar variety. You can now insert text like you would in a normal text editor. All the basic keybindings still work, be it ctrl+right arrow for navigating, as well as your standard editing keys like backspace and delete. I wouldn't recommend using these, however, as you would be overlooking NORMAL mode's editing capabilities. Generate some dummy text using Lorem Ipsum, enough so that it fills your screen (about 10 paragraphs should do), and then return to NORMAL mode by hitting the key. Next I'm going to cover the basic navigation. +

+ In Vim, deviation from the home row of your keyboard is highly discouraged by design. On a standard QWERTY keyboard, when positioned correctly, all the "arrow" navigation keys can be found beneath your right hand, instead, while in NORMAL mode, left is 'h', down is 'j', up is 'k', and right is 'l'. +

+ + https://leafbytes.com/assets/ss_nvim_04-d88cf393.gif + Demonstration of basic navigation in Vim + +

+ While this initial lesson is rather underwhelming, this basic feature of Vim reinforces a subtle, but powerful productivity standard, which is that your workflow should be keyboard-centric. I believe that coders/programmers should at least have a few (if not many) keybindings that they have memorized and engrained into muscle memory regardless of which text editor they use. It's simply more efficient than navigating solely with arrow keys, and there should be as little physical strain on the hands and wrists as possible while typing. The quicker you can accomplish your editing, the more time you have to devoting to programming and thinking hard about your next line of code. +

+ Very basic navigation out of the way, let's cover word traversal. While still in NORMAL mode, at any point on your dummy text, hit the key 'w', and then the key 'b'. 'w' traverses the document word by word, 'b' does the same but backwards. +

+ + https://leafbytes.com/assets/ss_nvim_05-22e9be78.gif + Demonstration of word traversal in Vim + +

+ Okay, now you're moving a bit faster, instead of letter by letter, you're traversing the document word by word. Next, we'll cover traversing an entire line. Navigate to the middle of one line and while in NORMAL mode, type the dollar sign symbol, '$', and then after taking a moment to notice the result in your editor, type the number zero, '0', and again, notice the results: +

+ + https://leafbytes.com/assets/ss_nvim_06-b9a7ff2c.gif + Demonstration of line traversal in Vim + +

+ These two commands lead you to the end of your current line with dollar sign, '$', or to the beginning of your current line with the number zero, '0'. +

+ Now you have a few options for quickly navigating your document horizontally, what about vertically? In NORMAL mode, while positioned more towards the top of your document than towards the bottom, enter 'ctrl+d', and notice the results. +

+ + https://leafbytes.com/assets/ss_nvim_07-a39d4c94.gif + Demonstration of vertical traversal in Vim + +

+ Essentially 'ctrl+u' and 'ctrl+d' are Vim's version of pageup, and pagedown. +

+ The next keybindings important to vertical traversal are that of 'gg' and 'shift+g'(capital G). While in NORMAL mode, enter both of these and notice their results. +

+ + https://leafbytes.com/assets/ss_nvim_08-c1214d4c.gif + Demonstration of entire document traversal in Vim + +

+ These keybindings navigate the entirety of your document. 'gg' will bring your cursor to the first line of your document, while capital G, or shift+g will bring you to the last line of your document. +

+ At this point you might be underwhelmed, as it all seems like a more esoteric way of navigating using alternative keys other than pgdown, pgup, end, home, ctrl+left arrow, ctrl+right arrow, etc. And you'd be right...if this was all Vim had to offer. +

+ While there is more to navigation than what I have presented here thus far, I will now move on to the basics of Vim Editing, which will hopefully further elucidate the power of Vim's modal editing features. +

+ For this example, please stay in NORMAL mode (if you hit 'i' at any time prior to this and entered INSERT mode, simply return to NORMAL mode by hitting the key). Navigate to the beginning of any word in your document using 'w'. From here enter, the keys, 'dw', and witness the results, yes, very simply, we deleted a word. +

+ I want to pause here and point out that in any other classic text editor, the keybinding to do this is ctrl+delete, which isn't so bad (though you do have to navigate both hands a bit far away from home row to do so). +

+ Also notice the chaining of commands here, 'd' is meant to be a mnemonic, reminding you of 'delete', but on its own doesn't do anything, it takes another command as an argument, so 'dw', means 'delete word.' We traverse using 'w' and when we come across a word we want to delete, we use the command 'd' and the argument, 'w', resulting in 'dw'. What if we wanted to delete the previous word? Well we traverse using back to the previous word using the 'b' keybinding, so it makes sense that 'db', would be 'delete back' or 'delete previous word'. +

+ + https://leafbytes.com/assets/ss_nvim_09-1c0c0366.gif + Demonstration of deleting a word using 'dw' and 'db' in Vim + +

+ And indeed it does. And yes, this functionality can be accomplished using the standard text editor's keybinding 'ctrl+backspace'. +

+ Should you wish to undo any of your recent deletions, simply hit 'u' to undo the last command. If you then change your mind and wish to redo your recent deletions, simply hit 'ctrl+r'. On standard editors, this redo functionality can be accomplshed usng 'ctrl+z'. +

+ You may still be unconvinced of Vim's editing capabilities, as I have covered how to accomplish many of these edits in a standard text editor as well. Allow me to hammer home the point by covering a few more editing techniques involving just the 'd'/delete command. +

+ Suppose I think a bit on how else I can chain 'd' with other movement commands? What if I wish to delete just the next character to the right of me? Well 'l' will navigate the cursor to the right, therefore d, 'delete', will delete, followed by 'l', which chains to mean 'delete one character to the right of me.' So 'dl' deletes one letter to the right of my cursor and 'dh' will delete one character to the left of my cursor. What about 'dj'? +

+ What about 'dd'? Well this one is less intuitive, isn't it? 'delete delete'. Well long story short, it deletes the entire line you are currently on. +

+ What about 'd$'? You guessed right, delete all text from my current cursor position to the end of the line. 'd0' will delete all text from my current cursor position to the beginning of the line. +

+ 'dG' will delete everything from my current cursor position to the end of the document, while 'dgg' will delete everything from my current cursor position to the beginning of the document. +

+ + https://leafbytes.com/assets/ss_nvim_10-52a5e17f.gif + Demonstration of deleting a word using 'dw' and 'db' in Vim + +

+ And so on... +

+ This concludes a basic exemplification on the power of chaining commands. Now, how would we accomplish these things in a standard text editor without this MODAL editing feature? Well this is where the ergonomics of your standard text editor fail. Let's just say I wanted to do more than delete a few words, what if I wanted to delete everything from my current position to the end of the line? Well in Vim, as previously pointed out, that is accomplished simply by invoking 'd$'. In a standard text editor, this would be accomplished by 'ctrl+shift+end' followed by 'delete' or 'backspace'. Four keys to Vim's two, with the latter example utilizing keystrokes that take your hands much further away from home row. +

+ At this point, you may protest, saying "Well I could always use the mouse to highlight that text and then just hit the delete or backspace key! It's not a big deal!!" And that is where you'd be wrong, were you to make such an argument. While I am not opposed to the utilization of the mouse in text editing (I am known to use my mouse from time to time), the frequency with which a programmer must switch from moving one hand away from home row on their keyboard to utilize the mouse in a standard text editor is something that I simply consider to be an inconvenience. Keeping your hands on the keyboard as often as possible simply is a best practice that should be reinforced when editing text. +

So Uhh...How Do I Quit?

+ It is a joke/meme among Vim users that many of us never leave Vim. Not necessarily because we love it, but because we never learned how to exit. Although the introduction screen does actually tell you how to exit Vim, it is easy for a beginner to forget the command after having experimented with it for the first time. +

+ From NORMAL mode, enter the colon key, ':'. Here you are entering "COMMAND" mode, which has syntax somewhat similar to the shell. If you wish to save the document you have worked on, enter 'w', which stands for 'write'. If you wish to quit without saving, enter ':q!'. The exclamation point at the end is akin to certain shell commands where '-f' is mnemonically associated with the word 'force'. Without it the editor will warn you that you have unsaved changes and not let you exit. Of course, you can chain these commands, so if you wish to write, simply enter ':wq'. +

Conclusion

+ I will digress at this point and bring this article to an end. There are far more Vim keybindings to cover, but I do not wish to overwhelm you too much, as the basic navigation/editing keybindings presented above are a good start to using Vim. In order to utilize Vim to actually start coding, however, will require at least a few plugins, and a bit of configuration. I do plan to eventually write an article covering these more nuanced aspects to Vim, but in the meanwhile I encourage you to take a look at the following official documentation on Vim. Thank you for taking the time to read my personal take on Vim, and Happy Coding! +

The Keys That Bind

Sun, 13 Nov 2022 12:30:00 +0200

the keys that bind

on tiling window managers

+ + https://leafbytes.com/assets/i3_logo-1988ad12.webp + The i3 Logo + +

+ "This allows the user to create a more lightweight and customized environment, tailored to their own specific needs." +
-The Arch Wiki
On Window Managers +

Introduction

+ When one first approaches the world of computers, their first interaction is with some kind of graphical user interface, commonly termed as a "GUI." This interface is what is termed as a "Desktop Environment" or "DE" for short. A common Desktop Environment's primary form of interaction is based around the utilization of the computer mouse, although, to varying degrees, can be modified to be more oriented towards a keyboard-based interface. +

+ Less familiar to the general public is a piece of software known as a Tiling Window Manager, a much more basic, but possibly more efficient, GUI that is heavily based on the utilization of keyboard interactions from the user. Indeed, much like the Vim Text Editor, a Tiling Window Manager so heavily discourages the use of Mouse Interactions, that an uninitiated user can find themselves presented with an environment where attempts at using the computer mouse results in no feedback at all (i.e., clicking the mouse does nothing by default). At first glance this may seem as a negative, but with the incorporation of more and more keyboard-encouraged interactions with your various pieces of software, one finds that their efficiency utilizing their computer increases, and perhaps also their productivity increases as well. +

What Is A Desktop Environment?

+ Prior to the advent of computer screens, all visual output from the computer was generally reserved to reading out lengthy printed pages via a line printer, while computer screens were reserved for keeping track of the status of the programs themselves. +

+ I won't go into the extensive history of computer screens, + window managers, tiling window managers, and desktop + environments, suffice it to say that the modern tiling window + managers and desktop environments are the products of the long + and arduous effort to bring computers into the modern home. +

+ The average modern desktop or laptop computer user finds themselves interacting with their Operating System via a Desktop Environment, clicking on icons via their mouse, scrolling articles in the Google Chrome Browser, perhaps typing a bit here and there when they have to perform a search query, or perhaps even typing a decent amount when they have to write an extensive document of some kind. +

+ The ability to type fast, maneuver between applications, context switch between different applications, and perhaps most importantly, the ability to make concise logical decisions, all play a role in how quickly a project is completed. Due to the vast amount of tools we have as modern computer users, and the multitude of ways of thinking and approaching a given task, there is, to put it simply, rarely a "one size fits all" workflow. +

+ The modern Desktop Environment found on Microsoft Windows is the most popular interface with which the average desktop computer user interacts. This is due mainly to the fact that they (the users) are unaware that there are alternatives (or that these alternatives are worth investigating). Even beginner programmers rarely take the time to investigate alternative Desktop Environments, limiting themselves to either working on Windows or MacOs and accepting the default keybindings and workflow instilled in them by the designers and developers of that Desktop Environment. They may investigate what kind of text editors are available to them, although this also is becoming increasingly rare as the majority of beginner programmers reach for VS Code and don't look into what alternatives are out there. This is somewhat unfortunate, as by taking the time to investigate these alternative tools, you may very well find more optimal solutions that better coordinate with your particular workflow. This article is to give you a brief introduction to how the Tiling Window Manager can play into a keyboard-centric workflow. +

The Keyboard-Centric Workflow

+ There have been few articles written on the keyboard-centric workflow. The subject is the realm mainly focused on by productivity enthusiasts and computer programmers (perhaps also the occasional stenographer). I have touched briefly on the concepts of the keyboard-centric workflow in my article on The Vim Text Editor, but will spend more time elaborating on the subject here. +

+ I have always enjoyed the sensation of typing from an early age when I first sat down at a keyboard. In high school and especially college, I took the time to learn some of the classic keyboard shortcuts present in nearly every application. The knowledge of these shortcuts allowed me to navigate more quickly throughout applications, edit text more quickly, and generally work with the computer much faster than others. In short, I became what is known colloquially as a "power user". +

+ It is intriguing to me, having now used Linux for a little over two years, to come to understand that the subject of keyboard shortcuts, or "keybindings," is a point of divergence within the varying philosophies amongst the developers of the major Operating Systems and their Desktop Environment Designs. Within Microsoft Windows and MacOs, one is heavily restricted to utilizing their specific keybindings. The standardization of keybindings, I'll admit, is somewhat essential so that when a user sits down at a colleague's computer, they are not completely lost as to which keys do what, as they are the same across all machines. Unix-like Operating Systems and their many Desktop Environments operate on the general philosophy that user freedom, and by proxy, customization options, is paramount. Thusly anyone operating a Linux or BSD based Operating System today is capable of setting up their own custom keybindings to their liking on their personal machines. This does make working from machine to machine difficult in terms of cohesion when working on a team, but by the same token, this freedom allows for a customizable work environment that is specific to its user. It is in my opinion that this is preferable as it allows the user the freedom to customize their Desktop Environment to their unique and particular workflow, which should, in theory, mesh well with their own personal thought process and problem solving techniques. +

+ Obviously, my preferences in Desktop Environment Design are geared towards creating a personalized keyboard-centric workflow that is unique to the regular user of that particular machine. The remainder of this article will focus on transitioning from a mouse-centric orientation with your computer towards a more keyboard-centric one, with the end goal of installing a Tiling Window Manager on Linux and the introducing the basics of utilizing it. +

A Brief Note On The Linux Desktop

+ As you may have already ascertained if you have read my Why Use Linux article, Linux allows for a great deal of user freedom and choice. Linux Desktop Environments are no different. There are a wide variety of options for Desktop Environments, each with their own features, advantages, and disadvantages. I must confess I only have extensive experience in one Linux Desktop Environment, that being XFCE. If you are curious, the other popular Linux Desktop Environments currently include GNOME, KDE, Cinnamon, and the LXQT Desktop environments. I cannot speak to the quality of these Desktop Environments, but would assume they have a similar amount of customization options to XFCE. +

Customized Keybindings On Linux

+ Most Desktop Environments on Linux have a GUI interface for changing the default keybindings, and I would encourage you to look into your particular interface for doing so. The Desktop Environment I utilized when I first started using Linux was XFCE, afterwards I then utilized the i3 Tiling Window Manager (and then the bspwm Tiling Window Manager, which is what I use currently). I will first go into how to adjust the keybindings in XFCE both via XFCE's native GUI application. After that, I will then go into how to utilize the i3, which is considered by many as a good introduction to Tiling Window Managers. +

+ Firstly, let's start by configuring the keybindings in XFCE. Here we see the base install of Debian Linux with the default XFCE Desktop Environment installed: +

+ + https://leafbytes.com/assets/ss_debian_xfce-b14da7cf.webp + The Default Debian XFCE Desktop Environment + +

+ The default terminal emulator in the XFCE Desktop Environment is called xfce4-terminal, thusly if we invoke it from the command line, another terminal emulator should spawn: +

+ + https://leafbytes.com/assets/ss_debian_terminal_spawn-186101b1.webp + A Picture of the XFCE Terminal Calling Another Instance of Itself + +

+ And indeed it does. So now we know we can call xfce4-terminal by typing out that exact name into the command line, but what if I want to call it using a custom keybinding? +

+ In XFCE's version of the start menu (located on the upper left hand side), you will find a menu item called "settings", where you will find the "keyboard" option. This is one of two GUI menus that allow for the modification of keybindings. Click on that so we can get started with one of the first recommended keybindings, that being of "super+enter", which we will bind to open our terminal emulator. +

+ + https://leafbytes.com/assets/ss_debian_xfce_keybindings01-8b32dcd3.webp + XFCE menu navigation to change keyboard shortcuts + + Once opened, Navigate to the Application Shortcuts sub-menu which should present to you some previously configured keybindings that are present by default. We are going to leave those alone and set up our own custom one by clicking on the "Add" button on the bottom left hand side of the menu. You will be presented with an empty input prompt asking for the command you will wish to invoke with this new keyboard shortcut. We want to open our terminal so let's enter the command 'xfce4-terminal': +

+ + https://leafbytes.com/assets/ss_debian_xfce_keybindings02-199b6eae.webp + XFCE menu navigation to change keyboard shortcuts + +

+ After hitting the "OK" button, we will be presented with a menu that asks us to "Press any Key". Most keyboard shortcuts invoked utilize at at least two buttons so as to not accidentally invoke commands whenever typing, so let's choose a classic keybinding "super+enter". To the uninitiated the "super" key is the "windows" key in Microsoft Windows or the "options" key in MacOs. Once entered, the window prompt will disappear and you may notice the new keyboard command show up on the Application Shortcuts menu. +

+ + https://leafbytes.com/assets/ss_debian_xfce_keybindings03-acc939b0.webp + XFCE menu navigation to change keyboard shortcuts + +

+ Let's test it out, go ahead and enter "super + enter": +

+ + https://leafbytes.com/assets/ss_debian_keybinding_01-a71368f6.gif + Demonstration of new Keybinding in XFCE + +

+ And indeed, we have created a useful keybinding that calls one of our most used applications. You can chain various keys to open applications, but usually the alt, ctrl, or super keys are utilized. You can call any application that you can invoke on the terminal in this fashion, and usually these are GUI applications, like your browser, your file manager, your music player, etc. +

+ I will now move onto our next subject, which covers the useful feature of workspaces. The use of workspaces is more important than application-related keyboard shortcuts as they essentially maximize the use of space on your screen, allowing you to bring up an entirely new work area for you to fill with your various applications. Instead of moving application windows around and minimizing them when not in use, Workspaces allow you to simply keep those applications open at whatever screen size you like, hiding them from view temporarily, while you navigate to another workspace to do some other sort of work, and then navigate back to your previous workspace to utilize the applications you had been using earlier. The below GIF demonstrates the use of Workspaces, which are changed using keybindings. +

+ + https://leafbytes.com/assets/ss_debian_keybinding_02-29e9b0ed.gif + Demonstration of Navigating Workspaces in XFCE + +

+ As you can see, each time I enter the super key followed by a number, it brings me to that numbered workspace. The change of the various applications is the change of the workspace, each one having different applications open. There are usually 10 workspaces available in tiling window managers. XFCE allows for 12 (using by default ctrl + f1-12 keys). In case you were wondering if I modified the keys for these, I did. That is because the super+number keybinding is more traditional in tiling window managers for switching workspaces. In XFCE, if you wish to change the workspace keybindings, navigate from the start menu to the settings > window manager sub-menu and select keyboard. You should then edit the keybindings labeled "Workspace" followed by the workspace number. This is basically the same process as editing keybindings for applications. +

+ I could continue to explore the XFCE interface, but let us now move onto the subject of Tiling Window Managers with the introduction of the i3. +

The i3 Tiling Window Manager

+ The installation of the i3 Tiling Window Manager is relatively straight forward, as it simply involves utilizing your native package manager to do so. It is highly likely that if you wish to install a Tiling Window Manager for the first time, that you'd like to keep your current Desktop Environment in the event that you don't like it, and wish to revert back. This is completely understandable, and thus our tutorial here will cover how to install i3 alongside XFCE so that you can simply choose one or the other at login. +

+ Let's get started with first installing the needed pieces of software. From your terminal emulator , install the i3 window manager, the i3status display bar, the picom compositor, the dynamic menu, dmenu, and also the feh image viewer, for setting our background wallpaper. +

+ On Arch Linux: +

[~]$ sudo pacman -S i3 i3status picom dmenu feh

+ On Debian Linux: +

[~]$ sudo apt-get install i3 i3status picom dmenu feh

+ Now we can simply reboot our machine, upon login, before entering your name and password, be sure to select i3 from the wrench icon available at the top right hand corner of our login screen: +

+ + https://leafbytes.com/assets/ss_debian_i3_01-c15fb4df.webp + Screenshot of i3 Selection in LightDm Menu + +

+ Immediately you will be presented with a far more minimal set up than XFCE. You will be prompted if you'd like to generate a default configuration for i3, go ahead and choose "Yes" and hit enter. +

+ + https://leafbytes.com/assets/ss_debian_i3_02-4d047981.webp + Screenshot of i3 Welcome Prompt + +

+ Afterwards which you will be asked which "Mod" key you'd like as default, either the "super" or "alt" key. This key will be the default for chaining with other keys which when used in sequence will call various applications and change workspaces. I personally prefer the "super" key, so I'm going to choose the first option. +

+ + https://leafbytes.com/assets/ss_debian_i3_03-f38213c6.webp + Screenshot of i3 mod key choice prompt + +

+ After this choice is made and you hit enter, you'll be left with no further instruction, which can be jarring for beginners. Don't worry, remember the keybinding to call the terminal we made in XFCE? Well the reason I chose "super + enter," was because that is the same command in the default i3 configuration for calling the terminal. So you can do the same here in i3, calling the terminal with "super + enter". In fact, let's call a few terminals to see the default tiling behavior of i3, hit "super + enter" a couple of times: +

+ + https://leafbytes.com/assets/ss_debian_i3_demo_01-143d061e.gif + Gif showing i3 calling terminals + +

+ As you can see, a Tiling Window Manager "tiles" its windows in a cascading fashion, utilizing as much of the available screen space (known in some circles as screen "real estate"). So how do we navigate this? Well you can indeed use the mouse to select which of your open applications you'd like to utilize, but obviously, this article being about the keyboard-centric workflow, we'll choose to navigate using our keyboard. Hitting "super+j" or "super+;"(super + semicolon) will navigate left and right respectively. Additionally, if we move a window to the bottom so that it is in a horizontal orientation using (super+shift+k), then we can demonstrate navigation up and down windows using (super+k) or (super+l) +

+ + https://leafbytes.com/assets/ss_debian_i3_demo_02-a9072a57.gif + Gif showing basic navigation in i3 + +

+ Already you can see the elegant simplicity this kind of workflow enforces. Utilizing some simple keybindings, one can navigate from one window to the next, all the while entering commands into a terminal, text into a text editor, and search queries into a browser. +

+ I won't be going into any kind of extensive coverage of customizing your i3 environment, suffice it to say that Tiling Window Managers in general are highly customizable. Instead, I will refer you to the excellent i3 documentation. +

+ As an aside, you may have noticed I also had you install dmenu and feh alongside i3 and its status bar. This is because the default configuration of i3 has a built in call to dmenu (super+d). This application is an excellent piece of software that, like the terminal, can call any application simply by entering its name into its prompt. I recommend playing around with it and checking out The Arch Wiki's documentation on dmenu. The image viewer, feh, has a nice feature of being able to set the background wallpaper to whatever image you'd like when i3 starts. In your ~/.config/i3/config file, simply enter on its own line the following command: +

exec feh --bg-fill /path/to/your/preferred_image.png

+ If you'd like to restart i3 without having to log out, simply enter super+shift+r(if you make any adjustments to your i3 configuration file, you won't see the changes implemented until you reload i3 using this command or you log out of i3). +

+ To log out of i3, enter super+shift+e. This will bring up a yellow message at the top of your screen asking if you'd like to exit i3, click 'Yes,' and it will return you to the login screen. +

Conclusion

+ There is so much more I could cover regarding Tiling Window Managers and the keyboard-centric workflow, but I will leave you with this basic introduction on how to get started with creating your own version of it. While a classic Desktop Environment is fine for most, and will probably always be more popular than Tiling Window Managers, for those of you that have an affinity towards working on the keyboard (which, if you are invested in becoming more productive on the computer, you should), a Tiling Window Manager might be something worthwhile for you to investigate. Below I provide you with a short clip of my standard workflow (if you're wondering how to get transparency to work in the terminal like in this GIF, you'll want to look into The Arch Wiki's documentation on picom). To further emphasize the use of the keyboard, I have installed the Vimium extension into the Librewolf browser to demonstrate further the possibilities of working almost completely without a mouse. Enjoy, and keep keybinding. +

+ + https://leafbytes.com/assets/ss_debian_i3_demo_03-2b892e61.gif + Demo of a highly customized i3 setup + +

Ortholinear Keyboards

Sun, 18 Dec 2022 12:30:00 +0200

ortholinear keybords

they're just better

+ + https://leafbytes.com/assets/OLKB_Planck-20119d22.webp + A 40% Planck Ortholinear Keyboard + +

+ "The orthogonal features, when combined, can explode into complexity." +
-Yukihiro Matsumoto
Author of The Ruby Language +

Introduction

If you've read any of my previous articles, you can probably tell that I'm pretty obsessed with The Keyboard-Centric Workflow. In utilizing the NeoVim Text Editor, The Vimium Browser Extension, and the bspwm Tiling Window Manager. I can honestly say that my potential productivity has greatly increased. Although I owe a great deal of this increased potential productivity to the aforementioned pieces of software, there is one piece of hardware that has been a game-changer to my workflow and that is the ortholinear keyboard. +

I first encountered an ortholinear keyboard after witnessing my mentor working with a variety of ergonomic keyboards with different layouts in an attempt to address the carpal tunnel he had developed over nearly nine years of programming as a software developer. At one point he pulled out what is known as a 40% Planck ortholinear keyboard. You can see an example of of such a keyboard at the beginning of this article. +

At the time I first saw one of these I was a little perplexed as it appeared to have no number or symbol keys (and in the case of my mentor, there were nothing printed on the keys themselves, so it was even more perplexing). I was more interested in learning the basics of Vim at the time, and had shelved investigating these odd looking mysterious devices until I had a better grasp on text editing and the syntax of JavaScript. At one point, however, on perusing reddit, I discovered a subreddit devoted to these interesting devices, r/olkb. +

I decided that, while I loved my current standard keyboard, that I was tired of reaching for keys, and that the 40% Planck ortholinear keyboard (OLKB) from Drop was a worthy investment. After a short two day period, I had memorized the new layout and was flying faster across my Tiling Window Manager than I ever had before. It solidified and dare I say, finalized, my keyboard-centric workflow. A Tiling Window Manager, A Vim Text Editor, Vimium in The Browser, and an ortholinear keyboard, these things didn't render the computer mouse obsolete, but did indeed heavily enforce use of the keyboard to the point where there was far less context switching between applications, and far less reaching away from home row. +

This brief article is an introduction to ortholinear keyboards. In it, I will cover why they are beneficial to anyone who spends a decent amount of time at their desktop computer, and the varying types available to anyone looking to invest in them. +

As a brief disclaimer, I must say that no one has sponsored me for this article, and the brands mentioned are solely there as a reference in a truthful account of my limited, but particularly positive, experience with ortholinear keyboards. +

Standard Vs Ortholinear

The Keyboard, as we know it today, has a long history, originating with the invention of the Typewriter. The standard of staggering of the keys diagonally out from the space key originated with the Typewriter in order to accommodate the mechanical linkage of the switches. This staggering of the keys was adopted by the computer keyboard, and persists on the majority of keyboards to this day. Many of us, myself included, learn to type on these staggered layouts, and rarely, if ever, question this choice in design. I would posit, however, having experienced and enjoyed the use of an ortholinear keyboard for a short time now, that perhaps you should. +

The orthogonal orientation and linear layout of the keys is what differentiates an ortholinear keyboard from the standard keyboards the majority of computer users are accustomed to. In orienting the keys in this way, the digits of the hand no longer "reach outwards and to the left" towards the periphery of the keyboard in order to access certain keys. Rather, the fingers "reach symmetrically both vertically and horizontally" which is a more natural movement when making small dextrous motions like typing. Additionally, ortholinear keyboards often incorporate other ergonomic design choices, such as reducing the overall size of the keyboard by utilizing other keys known as "layer" keys which allow the typist to access symbols and numbers through holding down what is akin to a second and third shift key. This allows for even less movement of the hands away from home row. There are also what is known as split keyboards, where the board itself is physically divided into two halves. Split keyboards are designed so that the typist may have their hands rest in a more natural position closer to the sides of their body (many of these split keyboards also incorporate the aforementioned design choices like ortholinear layout and the use of layer keys). +

A Brief Word On Layouts

There are an extremely wide variety of keyboard designs in use today, but the majority of them use only one layout, the QWERTY layout, with COLEMAK and DVORAK being relatively unknown alternatives with their own niche user base. These layouts are designed to be more ergonomic and can theoretically be utilized more efficiently (faster) than QWERTY. I'll admit I have no experience utilizing these other layouts and as much as I love the keyboard-centric workflow, will probably not explore these other layouts for some time, if ever. This is due to the fact that I personally believe that having comfortable access to the entirety of the physical keyboard has a greater impact on typing speed than the layout itself. This is because the time taken moving your hands up and down and away from home row may have a greater effect on typing speed than the key layout. This isn't to say that in the hands of an experienced COLEMAK or DVORAK user, an ortholinear keyboard can't be used more efficiently than a typist using that same ortholinear keyboard with the QWERTY layout. Rather, I am simply expressing my opinion that this increase in words per minute is probably negligible when compared to the benefits of simply converting to an ortholinear keyboard. I'll admit that currently, I'd rather invest in a split ortholinear keyboaord than invest in learning one of these other layouts, but I am always open to debate this topic if you'd like to reach out to me, perhaps you can change my mind. +

Should You Use An OLKB?

Obviously, my preference is to utilize an ortholinear keyboard, but just like with the previous topics I have covered, it is a matter of personal taste and workflow. The choice of keyboard, in particular, is a very personal one because unlike a piece of software, you are interacting with the keyboard physically, and very very often. Thusly what may be comfortable and enjoyable for me to use may not be so for you. That said, I do believe an investigation into utilizing some kind of specialized keyboard is worthwhile for any software developer or heavy desktop computer user. Programmers in particular spend an exorbitant amount of time in front of the computer and thusly investing in certain pieces of equipment that make that experience comfortable, efficient, and enjoyable, are paramount. This is not just limited to your choice of keyboard, but also office chair, computer screen, and other pieces of equipment. This may very well be even more important than choice of software, as a poor choice in equipment can, over time, result in poor posture and all the physical ailments that can result from that. Physically speaking, man was never meant to sit still hunched over a computer screen for hours on end. Thusly it is imperative that you ensure the least amount of harm to your body is incurred while you pursue your computer related endeavors. +

Conclusion

Unlike my other articles, which often offer up an introductory tutorial on the subject, this article will not provide you with any such introduction, as the concepts surrounding the use of ortholinear keyboards and the use of layer keys is rather self explanatory. If you do choose to purchase and learn to use an ortholinear keyboard, know that it may take a few days to a week to reorient yourself to it, depending on frequency of use. Keep at it, you'll find the benefit of learning to use your new keyboard to be worthwhile, especially when combined with use of Vim and a Tiling Window Manager. +

Although I'm sure that if you're intrigued by the subject, you'll do your own research, I have provided some images of various ortholinear keyboards below as a reference. Additionally I encourage you to watch this video from Ben Vallack, where he goes into depth covering the differences between standard and ortholinear keyboards. I should note that I have no personal or professional affiliation with Ben Vallack, but I've thoroughly enjoyed Ben's videos for some time now, and I feel like he covers this subject very well. +

+ + https://leafbytes.com/assets/olkb_01-343c22aa.webp + A Minimaliist Split OLKB + + + https://leafbytes.com/assets/olkb_02-1232d01e.webp + An Ergodox Split OLKB + + + https://leafbytes.com/assets/olkb_03-c09b69d8.webp + An Miniaxe 40% Split OLKB + +

Espanso

Wed, 11 Jan 2023 12:30:00 +0200

espanso

expand your text

+ + https://leafbytes.com/assets/espanso_logo-15ea8537.webp + An image of the espanso logo + +

+ "The texts we write are not visible until they are written. Like a + creature coaxed from out a deep wood, the text reveals itself little by little. + The maze evokes a multiplicity of approaches, the many tricks we employ to tempt + the text hither." +
-Ricki Ducornet
The Deep Zoo +

Introduction

+ Prior to a friend of mine mentioning espanso to me, I was unaware of such a + thing as a text expander. I was familiar with the idea of expanding text, and + until I adopted espanso, I was utilizing Neovim's configuration file to set + custom text expansion. For example within my init.vim (which I used instead of + lua at the time), I had set up a shortcut to write out "console.log()" whenever + I inputted "ccl" while in insert mode. +

+ I had many such shortcuts, mainly meant for use with various console calls in + JavaScript, as well as some generic coding shortcuts. At one point my friend had + pointed me in the direction of a new text expander written in rust known as + espanso, and today it has become a regular part of my workflow. Whether it be a + part of the command line, Neovim, or my browser, I utilize espanso in nearly + every aspect of my workflow now, and I felt it was worthwhile to cover in + detail. +

What Is A Text Expander?

+ Most standard workflows by "Power Users" involve the use of keyboard shortcuts. + Mainly Ctrl+C, Ctrl+V, Ctrl+A, and Ctrl+Z. If one is particularly resourceful, + they are aware of various keyboard shortcuts involving the additional use of ALT + and SHIFT keys, but I digress. While these shortcuts are still very useful and a + necessary tool in the belt of any serious Computer user, these methods still + only hold a candle to the ultra capable workflows one can achieve with a tiling + window manager, a modal text editor, and an ortholinear keyboard (as well as the + Vimium extension installed on the browser). Each of these subjects I have covered + in detail in previous articles, and each of them continually improves upon the + user's implementation of the keyboard-centric workflow. There is, however, one + other tool that greatly adds to the keyboard-centric workflow, and that is the + text-expander. +

+ Let's say you have a series of commands or parts of a document that you + repeatedly find yourself typing over and over again. Commonly known as + boilerplate, these repeated pieces of text can become cumbersome and are the + bane of any serious computer user, as this information is usually a formality, + and has little to do with the meat of the content of the document itself. Of + course, if there is a lot of text, one may find oneself simply copying and + pasting the text, or making a multitude of copies via some script. Let's say + there is some subtle difference between the documents though? Well using + standard UNIX utilities like grep, sed, and awk can accomplish what one wants in + an automated fashion. So where does that leave the text expander? With so many + other tools available to the modern power user like autocomplete in the shell + and text editor, as well as keyboard shortcuts like Ctrl+C/Ctrl+V in pretty much + every application, what more could you need? +

+ Indeed, when I first discovered that there even were applications like text + expanders, I found myself wondering why they even existed when one could write + aliases for your shell, and write custom shortcuts for NeoVim. After a short + period of time, however, it dawned on me the power of text expanders, and that + power was the ability to write quick aliases/text shortcuts that could be used + in any application. And when used in conjunction with shell scripts + and aliases, the power of text expansion becomes even more obvious. +

Espanso Basics

+ While I could go into the details of the espanso program itself, I'll simply + leave this link to the official + website for you to peruse. There are many other text expanders out there, but espanso + is open source and OS-agnostic. Additionally it is written in rust, which is known + to provide fast applications when utilized by a capable + programmer. It is the only text expander I have used, so I cannot speak + to how it compares to others, but considering the aforementioned features, along with my own + positive experiences utilizing the program, is enough reason for me to not investigate + the alternatives at the time of this writing. Should you wish to install espanso + and try it for yourself, please see their official installation + documentation. +

+ Using espanso is very simple. For example once you have the daemon running (see + the documentation), you can check your ip + address simply by typing in any application: +

[~]$ :ip

+ This will expand out to display your ip address! Of course, one could easily see this + using command line. Prior to using espanso, I had created a bash alias + to do just that (and admittedly it gives more information than espanso does by default): +

[~]$ alias ipaddr="curl ifconfig.me/all.json | jq"

+ Now, if this was all espanso ever did, I'd say the program was trivial and this + article wouldn't even exist. The power of espanso is not in its out of the box + settings, but rather in its simple config file. On Linux, this configuration + file can be found in: +

~/.config/.config/espanso/match/base.yml

+ Here, you will find the default text expansions under the section "match": +


+ matches:
+   # Simple text replacement
+   - trigger: ":espanso"
+     replace: "Hi there!"
+

+ And indeed if we type out the trigger ":espanso", we'll see our text is replaced + with "Hi there!": +

+ + https://leafbytes.com/assets/espanso_example-d90dca07.gif + Demonstration of Basic Usage of Espanso + + There are various default examples showing espanso's + utility. If you look towards the bottom of the + example text expansions, you'll see the ":ip" + example, which includes a very similar command to my ip alias: +


+# Returns public ip address
+- trigger: ":ip"
+  replace: "{{output}}"
+  vars:
+    - name: output
+      type: shell
+      params:
+        cmd: "curl ifconfig.me ; echo ''"
+

Writing Espanso Aliases

+ Now, all this is well and good, but what sort of things can one accomplish with + this tool? Well one of my most constantly repeated series of code that I need on + the fly is "console.log()" whenever debugging a JavaScript file. While this + isn't a lot of text to write, it can indeed get cumbersome to rewrite it many + times over, and so one my first espanso aliases was: +


+- trigger: ";cll"
+  replace: "console.log()"
+

+ Note that instead of using the colon, I opted to use the semicolon for my custom + espanso aliases as the likelihood of typing out anything directly after a + semicolon is even less likely than after a colon (especially if using Vim in + command mode). While this might seem trivial at first, if anyone has ever + debugged a JavaScript file using this most basic of tools, one can see how + useful this could be. +

+ Within JavaScript (or any programming language), there are + many times one can find themselves writing various words or + series of code blocks over and over again, here a few examples + of what I have written in my espanso config to occasionally speed this up: +


+- trigger: ";for"
+  replace: "for (;;) {}"
+
+- trigger: ";while"
+  replace: "while () {}"
+
+- trigger: ";if"
+  replace: "if () {}"
+
+- trigger: ";elif"
+  replace: "else if () {}"
+
+- trigger: ";else"
+  replace: "else {}"
+
+- trigger: ";$"
+  replace: "`${}`"
+
+- trigger: ";do"
+  replace: "do {} while ()"
+
+- trigger: ";func"
+  replace: "function () {}"
+
+- trigger: ";=>"
+  replace: "() => {}"
+
+- trigger: ";try"
+  replace: "try {} catch(error) {}"
+

+ Okay, so what about use outside of the terminal? Well in writing any website, + any good web developer will be typing either "127.0.0.1:" or "localhost:" a lot, + right? Well why not use this espanso alias instead?: +


+- trigger: ";loc"
+  replace: "localhost:"
+

+ Again, many of these seem trivial on the surface, and you might even be rolling + your eyes reading this thinking "Really? You can't be bothered to type out nine + characters vs four?" I'll admit time savings rarely come up to typing out only a + few less characters, but one of the powers of Vim is its ability to accomplish + more with less keystrokes, and when used in its entirety, I'd argue that no one + using the classic keybindings available in the majority of non-modal text + editors can accomplish the same speed of workflow as those who use some form of + modal text editing. Espanso moves these cumulative time savings into any + application where typing is a part of its interface! +

+ Let's move onto another use for espanso. I'm a big fan of + using DuckDuckgo's bangs. If you're unaware of bangs, check out the + website. + Essentially, if one has DuckDuckgo as their default search engine + (or searx, which has this feature too), one can + simply enter into your search/url bar: !yt and any other words typed after that + will be directly searched in YouTube. There are a multitude of these bangs + available and I'd encourage you to take a look at what is available. There are, + however, a few websites that DuckDuckgo hasn't integrated into bangs, so what to + do? While I'm sure there are other ways to accomplish this, I have utilized + espanso to do this for certain sites, like Odysee, a Youtube alternative: +


+- trigger: "!ody"
+  replace: "https://odysee.com/$/search?q="
+

+ Note my decision to use the exclamation point instead of the semicolon here, in + order to keep with the consistencies of DuckDuckgo's bang syntax. When I type + "!ody" into my url, this immediately expands to "https://odysee.com/$/search?q=" + and I can now type in my search query. This is even more efficient on the + utilized network than DuckDuckgo's bangs as it never even + queries DuckDuckgo before re-routing to the requested page. I also use this same + technique for other sites, like Quetre, a non-Javascript front-end for Quora: +


+- trigger: "!que"
+  replace: "https://qr.vern.cc/search?q="
+

An Interesting Use Case

+ Now, any other power user of Linux and Vim would note, "yeah, but I can easily + accomplish alot of these same things in the shell or vim itself, I don't need a daemon + listening in the background taking up (a little bit) of RAM for this." Which is + a fair point, but I'd argue that what one can accomplish with espanso integrated + into your workflow with the shell, vim, and the browser, is just a little more + streamlined than without. Now, this last and final example on the uses of the + espanso text expander will involve use of the shell, a shell script, and NeoVim. + I am positive there are other ways of accomplishing what I have done here, but + nevertheless I am happy with the result as it stands right now. +

+ For a while, I had been writing and rewriting a simple shell script that would save my + current working directory. Additionally the script would take an optional + argument of a file which would be utilized by a separate script or alias that + would return me immediately to the saved directory and open + the specified file when invoked. So, for example, if I wanted to save + my current directory with a file I was working on, I could call the script like so: +

[~]$ sdir myfile.txt

+ The script very simply saves an environment variable called $sdir, and an + optional environment variable for the file called $sdoc into a + dotfile. A corresponding alias, "rdir" could then return to said directory like so: +

[~]$ rdir

+ Originally, rdir was a shell script that looked for the existence of the $sdoc, + and if it existed, would call nvim on it, and if it didn't, would simply + navigate to the saved directory $sdir. This worked for some time, but I wanted + to be able to save a different directory to this dotfile and seamlessly return + to it, and the built-in command "cd", doesn't work in shell scripts. I thought + "No problem, cd works in an alias, I'll just source the file in the alias and + then use cd to move into that new directory." Only to find out that source + didn't work inside an alias... +

+ Like I said, there are probably other, less hacked together, ways to accomplish my goal + here. Nevertheless, it dawned on me that I could remove my rdir script and forego + using an alias by simply combining the sdir shell script with an espanso alias like so: +


+- trigger: ";rr"
+  replace: "source ~/.config/.sdrc && cd $sdir && nvim $sdoc"
+
+- trigger: ";rd"
+  replace: "source ~/.config/.sdrc && cd $sdir && ls"
+

+ Although the above commands solved my issues with both cd and source in the + shell, typing this entire command out each time I wanted to start working again + would have been cumbersome. Using the espanso text-expander daemon takes care of that for me! + This solution is definitely not as elegant as some might like, but it serves my simple purposes + (save my current directory and a desired file so I can easily navigate back to it, and also easily + save a different directory and file at any time). +

Conclusion

+ Admittedly, there are probably many other use cases for espanso I have not + explored in this article. If one looks over the documentation in more detail, + one will find a myriad of other useful tips more in-depth than the ones I have + presented here, so please take the time to check out their great documentation. + Now that I utilize espanso, I can't really imagine working without it. + It's like the last puzzle piece to my keyboard-centric work flow was filled in when I + discovered this amazing tool. +

+ To cap things off, I'll share with you the simple sdir shell + script below (as well as this link + on my GitHub). + Additionally, I'll provide you with some lines of Lua code that when put into + your + NeoVim configuration files, will return you to wherever you last saved in + your files (this works incredibly well with the aforementioned sdir script and + espanso aliases, as I simply return to the exact directory, file, and + place in that file I left off on simply by typing ";rr" into the command + line). Enjoy, and please do check out espanso, it's a great project! +


+#!/bin/bash
+# sdir.sh
+# used with espanso ;rr and ;rd command
+# A simple bookmarking script used in conjunction with rdir
+# to save a directory for easy navigation later
+
+# save our current working directory
+sdir=$(pwd)
+sdoc="${1:-''}"
+# allow overwriting of sdrc
+set +o noclobber
+
+# define and create .sdrc file
+sdrc="$HOME/.config/sdir/".sdrc
+if [[ ! -f "$sdrc" ]]; then
+    /usr/bin/touch "$sdrc"
+fi
+
+if [[ $# -gt 0 ]]; then
+    sdoc="$1"
+else
+    sdoc=""
+fi
+
+echo 'export sdir='"$sdir" >"$sdrc"
+echo 'export sdoc='"$sdoc" >>"$sdrc"
+

+ +


+-- init.lua
+-- jump to last place visited in file
+vim.api.nvim_create_autocmd('BufReadPost', {
+  callback = function()
+    local mark = vim.api.nvim_buf_get_mark(0, '"')
+    local lcount = vim.api.nvim_buf_line_count(0)
+    if mark[1] > 0 and mark[1] <= lcount then
+      pcall(vim.api.nvim_win_set_cursor, 0, mark)
+    end
+  end,
+})
+

Git Basics

Wed, 11 Jan 2023 12:30:00 +0200

git basics

understanding version control

+ + https://leafbytes.com/assets/git_logo-d785b39e.webp + An image of the git logo + +

+ "I'm an egotistical bastard, and I name all my projects after myself. + First 'Linux', now 'Git'" +
-Linus Torvalds +

Introduction

+ Whether working in data science, software/web development, or even if one is + just a modern power user of computers in general, one is probably at least + somewhat familiar with the Git version control system, or at least with the now + infamous hosting service, Github. +

+ As a beginning software/web developer, I often find myself overwhelmed by the sheer + vastness of the amount of tools available, and the intrinsically complicated + ecosystem in which modern software development takes place. There are few pieces + of software on which there is a ubiquitous standard in place like there is with + Git. In this article, I will attempt to introduce the Git version control system + and its most basic features. The intended audience of this article is a complete beginner to + Git. Thusly, this article will not cover some of Git's more in-depth features, + but please take a look at the provided links for more documentation at the + end. +

A Brief History of Git

+ In looking over my previous article covering Why Use Linux, I now realize I + never referenced the history of Linux and its creator, Linus Torvalds. This was + probably due to the extensive history of Linux, and I didn't wish for that + article to turn into more of a history lesson than a beginner's installation + tutorial. While it is not my intention to do so with this article either, I feel + that Git's history is far more easy to cover, as it is a more recently developed + piece of software and has a far shorter timeline to cover. +

+ Git was originally developed by Linus Torvalds in 2005. Torvalds, as I mentioned + earlier, is the creator and current maintainer of the Linux Kernel. Linux in its + initial days of distribution, was passed around by software enthusiasts using + floppy disks and later on CD-ROMs. Updates to the Linux Kernel in the early days + were distributed via email mailing lists. The emails included files known as "patches", + which provided the differing lines of code which were either added, removed, or + altered from the pre-existing code base. While some form of version control had been a part of + software development since the early 1960s, concurrent version systems would + become the standard for decades starting in 1975, which would later be overtaken + by Apache's Subversion system, and finally in the late 1990s, distributed + revision control systems would come to dominate the preferred versioning control + system by the majority of software developers. +

+ BitKeeper was the first of these distributed version control systems, and was a + boon to the Linux Kernel development team, who adopted a beta version of + Bitkeeper in 1999. There was some controversy surrounding the use of BitKeeper, + which at the time was a piece of proprietary software, and many + within the Linux community felt that this was in conflict with the principles + under which Linux was founded. In April of 2005, BitMover + (owner of BitKeeper) announced that it would stop providing + a free version of their software to the community. Linus Torvalds then decided + to take a couple of weeks off from development on the Linux Kernel to create a + new distributed version control system which he later called + "Git", named after the British slang for a "stupid person." +

+ Although originally not intended to be a full-blown version control system, Git + eventually became more than just a simple handful of + scripts, and is now utilized by the majority of software developers today. +

What Is Version Control?

+ To those uninitiated into the ecosystem of modern software development, version + control is pretty much what it sounds like. You have a piece of software you + have written, and you have posted it up online for all to enjoy (because you + released it under an open source license, right?). Eventually someone uses it + and discovers a bug, or has a feature request, or simply wants to take your project + and make their own version of it, but how do you do this quickly and easily? +

+ As I covered in the introduction, in the early days there wasn't a very easy way + to do this. You emailed a patch file to an email mailing list, and everyone who + utilized your software had to be subscribed to that mailing list and manually + patch the files themselves (oftentimes this process was partially automated via + makefiles or bash scripts). Thank goodness today we have distributed version control + systems like Git that largely make this process incredibly easy by comparison. +

Why Version Control is So Important

+ Let's make our example scenario from before more simple for the sake of + explanation. Let's say you write some basic JavaScript code like so: +

console.log("hello");

+ You save it to your local machine in a file called 'hello.js'. Eventually you + realize the triviality of such a program, and want to extend out its + functionality. For the sake of brevity, I'll not write out more code, but let's + say you write a series of functions and the total lines of code (LOC) goes from + being 1 to 40 lines. You intelligently save your work and also decide to back it + up on an external hard drive, and even have a friend back up the current version + of your software on their server at their house. Great! +

+ But the ideas keep coming, you keep on coding and adding features, occasionally + refactoring the code to be less verbose and more succinct. You're becoming a + better developer and the program is starting to look good. Eventually you decide + to integrate a large new feature and start to write out a substantial amount of + code. Once completed, you save your progress, back it up on your external hard drive + as well as at your friend's server. Satisfied with the new feature, you decide to call + it a night, and hit the hay. +

+ In the morning you return to your project only to find that this new feature has + broken a different part of your program. You accidentally didn't + compartmentalize the features, resulting in spaghetti code. It'll take you two + days to remove all the code you wrote yesterday, and the chance of introducing a + bug during this refactor is high. If only you could just roll back time and go + back...to a previous version. +

+ This simple example is but one of the many reasons why version control is so + important to software development, and doesn't even include the benefits when + working with multiple developers on a single project. So how would this scenario + play out had you used Git? Let's take a look on how to do that in the next + section. +

Getting Started On Github

+ Again, we write out our basic example code: +

console.log("hello");

+ We save our file, 'hello.js,', but this time we decide to use Git to save our + progress on Github. I won't be covering how to set up a Github account, but here + is + a tutorial on how to do so. +

+ Once you've established a Github account, log into Github, click on the "New" + button in the top left-hand corner. Set up a project name called "my_awesome_js_project" + and give it a brief description before hitting the "Create Repository" button at the bottom. +

+ After this, you will be presented with a project screen that is a visual + representation of your project. This initial screen is pretty bare bones, but + provides you with a helpful series of commands to enter into your terminal to + start using Git to version control your project. +

+ + https://leafbytes.com/assets/ss_github_01-6b8c17cf.webp + Initial Github Repository Page + + While you could technically start creating files and pushing them + to Github using their GUI interface here in the browser, that was + never the intended way of working with Git, which is a command line tool. +

+ So, returning to our initial example, we have a 'hello.js' file, in a project + directory under the same name as our project name from before, + "my_awesome_js_project". Let's initialize a Git repository inside of this + directory by typing into the terminal: +

[ ~]$ git init

+ Great, you should receive a brief message that covers the initialization of your + project. If you look around using the "ls" command, however, you'll find that + there aren't any new files...or are there? If you then try the "ls" command with + the "-a" flag, you'll see that we now have a hidden ".git" directory. You + shouldn't need to access this directory as a beginner, but it's good to know + it's there in case you do need to make some direct modifications to your Git + configuration. +

+ Continuing on, let's add some basic files to our repository. While we do have + our "hello.js" file, it's always a good practice to have a README.md file in our + repository. Github will present this README to anyone who navigates to your + project via Github, and it's generally a good idea to provide some basic + documentation about our project to those who take a look at it. Let's create a + very basic README.md via the command line per the suggestion on Github like so: +

[ ~]$ echo "# my_awesome_js_project" >> README.md

+ This will create a simple header in a README.md file, which you should now see in + your "my_awesome_js_project" directory alongside your "hello.js" file. Now let's + use Git to add these files to the repository. +

[ ~]$ git add README.md hello.js

+ Now, we're going to "commit" these files to our repository. You can think of + this as a kind of "staging" of our files. +

[ ~]$ git commit -m "first commit"

+ Technically at this point, the next suggested command is optional, but it is + very good practice to change branches. A brief explanation of what branches are + is necessary to the uninitiated here. One of the most powerful aspects of Git + and other versioning systems is its ability to keep track of multiple "branches" + of your project. Much as the name implies, you can think of branches as + divergences along a tree of versions of your project. Remember in our initial + example that you added an extensive feature that once written broke the rest of + the program? Well that is a good example of when a separate branch would have + been helpful, as once you were done working on that branch of the project, it + would not have been part of the "main" or "master" branch of the project. You + can think of the "main" branch as being the trunk of a tree, the major body of + representation of your project. When adding a new feature, fixing a bug, or + doing any sort of work where you don't wish to change the current code base, + it's always a good idea to start a new branch. +

+ Whenever a new git repository is initialized, it defaults to a branch known as + "master". This branch, in practice, should never be touched except by the + project maintainer, who will "merge" the project branches first into a + development branch commonly known as "main." Upon completion of certain + versions, the project maintainer will merge these changes into the "master" branch. + This is why Github's documentation instructs us to create a main branch like so: +

[ ~]$ git branch -M main

+ Finally, we'll configure Git to send our changes to our Github repository using + the following command: +

[ ~]$ git remote add origin https://github.com/'your_name'/my_awesome_js_project.git

+ This saves your remote Github repository in the .git directory's config file + allowing you to quickly send your changes to Github. Let's send it off to Github now: +

[ ~]$ git push -u origin main

+ You'll be asked to enter your login credentials. Github no longer uses passwords + and instead requires use of either a Personal Access Token(PAT) or an SSH key to make + changes to your repository. The set up for these are well documented, but can be + somewhat intimidating for a new user. While I recommend setting up an + + SSH key, you can also set up a + PAT for this purpose as well. +

+ Once you've entered your credentials, a short message letting you know of your + successful push to your repository will display in your terminal emulator. + Refresh the repository page on Github and you'll see that your project is now up + and running. +

+ + https://leafbytes.com/assets/ss_github_02-f2136b66.webp + Initial Repository Instantiation Page + +

+ Excellent, you have successfully instantiated your first Github repository. Now, + on the surface, this can appear just like a glorified backup system, and while + Git can be used for backups, the main feature of using Git + is for version control. Let's add some code to our "hello.js" file to demonstrate. + Open up the "hello.js" file in your favorite text editor and add the following line after + our initial console.log call: +


+console.log("hello");
+console.log("this is my second line of code to commit");
+

+ Any good piece of version control software will immediately recognize this + change to your code. You can confirm this by running a + status check via Git: +

[ ~]$ git status

+ Upon entering this command, Git will now provide you with a brief summary of the + changes to your project. In this case we have "modified hello.js". We have yet + to have added our changed files, committed our changes with a brief message + about what we did, and pushed our changes up to our repository, so let's do all + of that now: +

[ ~]$ git add hello.js

+ Note that we are only adding the files we've changed. We never touched the + README.md and thusly do not have to add that to our commit. Next let's write a + helpful commit message: +

[ ~]$ git commit -m "Added a second line as an example"

+ Here, I'd like to provide a brief word on best practices regarding commit + messages. A good commit message should be a short, concise pieces of documentation + that informs others working on the project (or just your future self) of what changes + were made. These messages should be descriptive enough to cover a specific change + that was made, without being overly verbose (less than 50 characters is a good + rule of thumb when writing commit messages). +

+ Finally, let's push our changes up to Github: +

[ ~]$ git push

+ Note that you no longer have to provide the remote name, nor the branch + name here. Github will once again prompt you for your credentials (either SSH + key or PAT). Once entered, you will get a confirmation message in your terminal + emulator, and can visit your Github repository to see the changes online: +

+ + https://leafbytes.com/assets/ss_github_03-9689d6e0.webp + Github repository now shows updated changes + + As you can see, our update is reflected next to our hello.js file on our + repository page. At this point, you will have a very basic understanding about + how to use Git. Now I'll cover some other useful aspects. +

Git Log And Reverting Back

+ You can see changes pushed to your Git repository locally by + invoking Git's builtin log: +

[ ~]$ git log

+ Invoking git log will show you the history of the project for every commit you make. + This output is useful should you wish to revert back to one of your previous + commits, or simply wish to review the history of a project. + Since you only currently have two commits, let's cover how to simply revert back + by one commit using the information presented here at git log. +

+ Next to each commit you'll see a long series of letters and numbers. This is a + unique identifier for that particular commit. Copy this unique identifier for + your initial commit using your mouse to highlight the identifier, and invoking + "CTRL + SHIFT + C". This will save the identifier to your clipboard. +

+ Now enter into the command line: +

[ ~]$ git reset --hard 'unique-identifier-goes-here'

+ Keep in mind that this will only change your local repository, and nothing will + have changed on your remote over at Github. If you wish to revert the remote + repository as well, you can do so by using git push with the -f or --force + option: +

[ ~]$ git push --force

+ Keep in mind that using --force is considered to be somewhat overbearing, and + you should always heavily consider the ramifications of using this option on a + collaborative project, as you can accidentally overwrite another contributor's code. + This is why when working on non-trivial projects it is always a good idea to + + create a different branch. +

Conclusion

+ Whatever your preferences regarding text editors, desktop environments, or + operating systems, using Git as a version control system is an essential skill + for the modern day software developer. Even though there are other version + control systems out there, Git has become ubiquitous with + version control. It is essential that if you are just starting out with + software development that you become familiar with Git and its many features, as + they will inevitably become a part of your daily workflow. +

+ As an aside, there are far more features of Git than what is covered here in + this article, so please see the + official documentation + for more information. Additionally, there are many pieces of software that make + working with Git less cumbersome than what is available at the standard command + line. Some of these projects include + + lazygit, Github's offical CLI tool, and for those of you + who use Vim, there is + Vim-Fugitive. +

+ I would be remiss not to mention my own CLI tool that I wrote in bash, called + bgit. While not nearly as fully featured as the + aforementioned software, it is a very basic wrapper around Git that automates + away some of the commands covered in this article. I wrote this script as + a way to teach myself a little more about Git and bash, and it is certainly not + without its flaws, but take a look if you're interested. +

+ Lastly, I'll encourage you to first become familiar with the + basics of Git before going to one of these other pieces of software, as understanding + how Git works from the ground up is essential before moving on to + utilizing these other tools. I wish you well in your journey towards better + software development and I hope I have helped you understand a little bit + more about this essential piece of software. +

The README

Fri, 13 Jan 2023 12:30:00 +0200

the readme

documentation for beginners

+ + https://leafbytes.com/assets/markdown_logo-53c5ed00.webp + An image of the markdown logo + +

+ "Documentation is a love letter that you write to your + future self." +
-Damian + Conway +

Introduction

+ Documentation in the world of software is an essential part of the development + process. Without proper documentation both users and developers are given the + monumental task of parsing out a project line by line in order to understand how + to debug the project, contribute to the project, or indeed, even utilize the + software itself. +

+ There are many forms of documenting projects, be they man pages accessible via + the terminal to official documentation provided by the developer hosted on their + website. One essential type of documentation that a new software developer + must become familiar with is the README.md file. If you have ever + perused a properly maintained repository on Github or other hosting + service websites, you have most certainly come across a README.md. The finished + document looks something like this: +

+ + https://leafbytes.com/assets/readme-pic-01-cf144d08.webp + A sample page of a readme + +

The Basic Readme

+ The README is a document that one can think of as being the entry point to learning + about and using a piece of software. Oftentimes this document provides + instructions on installation, basic usage, and configuration. Sometimes multiple + READMEs are provided on more extensive projects to help developers extend, + debug, and manage a piece of software. +

+ Not including a README, even when working on a trivial or basic project, is + considered poor etiquette, as having no README leaves any potential user or + developer with absolutely no information about what lies within the project, and + indicates that the project maintainer doesn't care about the project, or at + least, doesn't care to provide documentation. Very simply, it is always a good + idea to provide at least a basic README when creating a project. +

+ Although a README can be written in plain text or even HTML, the markdown + language has become the standard for writing READMEs, as it provides a quick way + to create formatted text that is well presented to the reader. READMEs are + traditionally capitalized so that the 'ls' command prioritizes its display, and + the .md appended to the end of the README indicative that the document is + written in markdown. +

The Markdown Language

+ Anyone familiar with The HyperText Markup Language, commonly known as HTML, will + find themselves comfortable working in Markdown, as many pieces of Markdown are + interchangeable with classic HTML elements, just expressed differently. +

+ In certain circumstances, the Markdown language is even more powerful than HTML, + as it can very quickly represent tables, graphs, checkboxes, and other common + presentation elements in a way that is easier to read and requires fewer typed + characters to express them. +

+ For the remainder of this blog post, I will cover how to create a standard README + for a software project. While Markdown has many extensive features such as + presentation graphs and tables, I will solely be covering the basics for the + sake of brevity and this is only meant as an introduction. Should you desire to + explore what other features the Markdown language has to offer, I advise you to + take a look at the official guide. +

Getting Started

+ At the end of the day, Markdown is simply an extension upon a plain text + document, so one can get started working in it right away. Navigate to the base + of a project you'd like to write a README for (or just create a test directory), + and let's create one from the command line using the touch command: +

[ ~]$ touch README.md

+ Next we'll open up our newly created README.md file in our favorite text editor + and start documenting our project. I'll be using NeoVim for my examples, and + will also be using an extension called + markdown-preview. If + you are reading this blog post and want to follow along in + VS Code, there is equivalent functionality available for that editor as well + in the form of different extensions. +

+ Let's write out a header, with the title of our project. I'm going to call my + project, "markdown_ex". +

+ + https://leafbytes.com/assets/readme-pic-02-be84a5af.webp + a h1 header displayed in both neovim and the browser + + As you can see, a single hashtag "#" character is interpreted by Markdown as + indicating the beginning of a header. This is equivalent to an

tag in HTML. + Let's create some other headers as an example. +

+ + https://leafbytes.com/assets/readme-pic-03-f2ec3447.webp + a series of headers expressed in markdown + + Now, what's nice about Markdown is at this point you can simply start typing out + your documentation. Unlike in HTML, there is no need to encapsulate everything + in paragraph tags, so working in Markdown becomes more akin to working in an + extremely basic word processor. You can revert to using HTML elements if you'd + like, but especially in this mentioned example, it is not necessary: +

+ + https://leafbytes.com/assets/readme-pic-04-4ccd74c8.webp + basic display of paragraph text in markdown + +

+ Let's now explore some of the basic features that Markdown has to offer. Let's + say we wish to stylize our text a little beyond just making headers. Well, we + have many options at our disposal. Let's create some bold, italic, underline, + and strike-through pieces of text: +

+ + https://leafbytes.com/assets/readme-pic-05-0f464300.webp + various stylizing of markdown text + +

+ Here we see the various syntax utilized for styling pieces of text in markdown. + I'll cover briefly the basic syntax here: +

**two stars creates bold text**
+__two underscores also creates bold text__

*single star creates italicized text*
+_single underscore also creates italicized text_

~~two tildes creates strike-through text~~

<>html style u tags create underline text<>

Creating Lists

+ As I mentioned earlier, the syntax of Markdown can be far nicer than HTML in + many respects. A good example of this is creating ordered and unordered lists. In + HTML, creating an unordered list looks like this: +


+    My first item
+    My second item
+    My third item
+

+ As opposed to in Markdown: +

- My first item
+- My second item
+- My third item

+ The following picture demonstrates the simple creation of ordered and unordered + lists in markdown: +

+ + https://leafbytes.com/assets/readme-pic-06-b264e549.webp + a demonstration of creating lists in markdown + +

+ Markdown is especially easy on the eyes when it comes to creating links. In + HTML, creating a basic link can be created like so: +

<my example link<


+                
+                    The same can be accomplished in Markdown like so:
+                    
[my example link](https://www.example.com)
+                
+
+                Code Snippets
+                
+                    The Markdown language is multifaceted, it can be utilized to document anything
+                    from scientific papers to simple blog posts and is often utilized in online
+                    forums. One of the most common usage for Markdown is code snippets, which can be
+                    expressed using backticks like so:
+                    
```console.log('nice code block!')```
+                
+                
+                    Or Like so:
+                    
`console.log('nice code block!')`
+                
+                
+                    If, for example, you needed to present some basic JSON, you could do so very
+                    simply like so:
+                    
``
+{
+    "firstName: "John",
+    "lastName": "Smith",
+    "age": 25
+}
+``
+                
+                
+                    The following picture demonstrates the effect:
+                
+            
+                https://leafbytes.com/assets/readme-pic-07-e7acff6e.webp
+                demonstration of writing code blocks in markdown
+            
+                Checkboxes
+                
+                    One of my personal favorite uses of Markdown is to place a list of checkboxes
+                    towards the bottom of my READMEs. This usually is a list of various aspects of
+                    my software that need to be worked on, such as new features, bug fixes, and
+                    documentation.
+                
+                
+                    Checkboxes can be instantiated like so, one is checked off, the other is not:
+                    
- [x] This is a completed task
+- [ ] This task has yet to be done
+                
+                
+                    And here is an example Picture showing the results:
+                
+            
+                https://leafbytes.com/assets/readme-pic-08-52d67547.webp
+                demonstration of how to write checkboxes in markdown
+            
+                Conclusion
+                
+                    The capabilities of Markdown are vast. In addition to what has been covered here,
+                    one can create tables, graphs, and embed images/GIFs into Markdown files to
+                    further help document your projects.
+                
+                
+                    A good README can often be the saving grace of a project. Even if a project is
+                    poorly designed or not well maintained, a good README can provide a general overview
+                    of a piece of software and help others navigate its use, configuration, and
+                    further development. Without a README, the potential user or developer is left
+                    solely with the code itself to determine everything about the project. Indeed, a
+                    project without a README is probably not worth investigating, unless the
+                    codebase is particularly small and even then, some form of introductory
+                    documentation would be highly encouraged.
+                
+                
+                    As a beginning software developer, the README is one of your most important
+                    resources when learning about a new tool, or even referring back to one of your old
+                    projects. It is within your best interest to develop a good habit of writing
+                    READMEs and other forms documentation for your projects, so take the time to
+                    familiarize yourself with one of the most common forms of it, Markdown.
+                
+                
+                    I do hope this has been helpful to you and provides you with a basic
+                    understanding of the Markdown language and how to utilize it for a basic README.
+                    Cheers and Happy Coding!
+                
+



Feeds: The Only Civilised Way to Read Online
Sun, 08 Jan 2023 19:47:21 -0500
How do you follow your favourite online creators and their blogs, articles,
+videos, podcasts, comics, or what have you? How do you choose what to read
+when you have a few minutes to kill on the bus, or when you want to get caught
+up in the morning with a cup of coffee?
+
+Most people hand this choice over to social media, inviting along the whole
+associated host of problems like clickbait; outrage amplification; snooping
+targeted advertising; radicalising rabbit-holes; echo-chambers and filter
+bubbles; algorithms choosing what to show you based on “engagement” rather
+than what you’d want for yourself; and on and on.
+
+There’s a better way — and there has been for decades! Amazingly, it seems
+underused even within tech circles, and almost completely unknown to the
+general public. It’s super easy to use, actually more convenient than
+social media apps, and leaves you in complete control of what you see.
+
+I’m talking, of course, about RSS/Atom web feeds, and I contend that they
+are not only a better alternative, but in fact I’d go so far as to say that a
+feed reader is the only tolerable and civilised way to read online! The system
+works really well and more in line with what (I think) most people actually
+want; it minimizes the use of harmful social media platforms; and it helps
+foster a more vibrant, independent, creative, and non-commerical Web. So drop
+your non-chronological algorithmically-obscured sponsored timeline, and let’s
+have a whirlwind overview of what feeds are and how to use them!
+What are feeds and feed readers?
+
+The core of the problem is this: You might have hundreds of people whose
+work you’re potentially interested in, but you can’t realistically regularly
+check whether each one has published something new recently. The solution
+(first cooked up around 1999!) is a web feed or news feed
+(sometimes RSS feed or Atom feed, for uninteresting technical reasons), combined
+with a program called a feed reader (or a news reader or
+aggregator). I strongly suspect that the wide range of inconsistent
+names has been a big hindrance to these tools being widely adopted, but in
+practice which name you use doesn’t matter: I’m just going to call them
+feeds and feed readers from now on.
+
+A feed is a machine-readable version of new posts on a site. For a blog, it
+might contain, say, the last 10 blog posts, much like the human-readable
+“front page” of the web version would. A feed reader is a program that
+monitors all of the feeds from all of the sites that you’re interested in, and
+aggregates new updates in one place for you to read at your leisure.
+
+So, if there’s a blog or podcast or something else you’d like to follow,
+you simply subscribe to it in your feed reader, and now all new updates will
+be automagically conveyed to you as soon as they come out. Your feed reader
+keeps track of what you’ve not yet read, and displays all posts in a simple
+chronological timeline, so you never miss anything. You can catch up whenever
+you want; there are no algorithms boosting controversial or sponsored
+nonsense; and no ads.
+
+Feed readers would traditionally have been desktop programs that downloaded
+all the feeds you were monitoring onto your computer. That model still works
+just fine (and in fact may already be available on your computer thanks to it
+being a built-in feature of the Thunderbird
+and Outlook
+email clients), but nowadays most people prefer to use online services so that
+their read/unread status for individual posts can be synchronized between
+their desktop and their phone. Articles can then be read in the browser or in
+a smartphone app, and you’ll always be up-to-date with what you’ve already
+seen and what you’ve not yet read.
+
+Which feed reader you choose doesn’t matter much. Some popular web app
+options include NewsBlur, Feedly, Feedbin and Inoreader, or if you’re up for
+self-hosting, Tiny Tiny RSS. Desktop
+applications include NetNewsWire, FeedReader, Newsboat, and Fluent Reader. There are many more;
+these are just a sample. All of the (non self-hosted) web services are paid
+products — that’s the tradeoff for not being profiled and advertised at.
+Fortunately, most have free tiers or free trial periods, so you can try them
+all out and see which you like best. Many try to differentiate themselves by
+offering extra features such as optional filtering or even
+machine-learning-based prioritization of your feeds, but in my case that’s
+what I’m trying to get away from so I just turn those features off
+and don’t use them. I personally use NewsBlur, which has worked just fine
+(along with their mobile app) for years.
+
+So, you pick a feed reader to try out, and set up an account. Then, you
+find the feeds published by sites you’d like to follow, and subscribe to them
+in your feed reader. You can then check in whenever you’d like, and any new
+posts will be there waiting for you in your reader. No more random
+outrage-bait or advertising, just a chronological feed of what you
+chose to follow.
+
+But does anybody actually publish feeds?
+
+Surprisingly, yes, almost every blog or publishing platform out there does
+in fact publish a feed (often automatically, without the author needing to do
+anything special to set it up). This includes publishing systems like Medium,
+Substack, Wordpress, and Blogger, as well as smaller self-hosted site
+generators. You can follow feeds from an astonishing variety of sources,
+including YouTube channels, Environment
+Canada weather warnings, CNN or the BBC, GitHub releases, status pages for online
+services, and of course, the myriad of small blogs out there on the web.
+The entire podcasting ecosystem is secretly built on top of feeds — that’s how
+new episodes get delivered to you! And, naturally, my feed is right here.
+
+To find a feed for a particular site, you can almost always just put in the
+address of the website into your feed reader, and it will auto-discover the
+feed. If not, first look for the ubiquitous orange and white feed icon: , perhaps alongside words such as “feed”, “RSS”, or
+“Atom”. There will be a link, which you can put into your feed reader to
+subscribe. Web browsers used to highlight the existence of feeds with an icon
+in the address bar, and you can restore that functionality to Firefox with this
+addon.
+
+What about discovering new authors?
+
+This is where some forms of social media perhaps have a defensible role to
+play. Potential sources for content discovery can be thought of as lying along
+a spectrum that at one end has high quality content but low frequency of
+discovering new sources, and at the other end has the incredible torrent of
+material that can be found online, which is usually of low value.
+
+The most reliable, but slowest, way of finding new feeds to follow is to
+wait for friends, family, or coworkers to share links to things they found
+interesting. Rather than just read them and move on, take a minute to poke
+around the site and see if you think you’d like to see more from that creator.
+If so, check whether they have a feed, and subscribe to it. If it turns out
+they never post again, there’s no harm done. If it turns out their future
+posts aren’t very interesting, that’s ok: just unsubscribe.
+
+A higher-volume but perhaps more inconsistent source of feeds to follow
+might be topic/community-focused public aggregator sites. These are a form of
+social media, but they center on particular subjects or communities, rather
+than whatever anyone would like to share. For example, technologists often
+follow Hacker News, Lobsters, or the Programming subreddit.
+Like with links shared with you directly, if you find something here you like,
+poke around the author’s homepage and consider following them for more. As a
+bonus, if you care about accruing karma/points/reputation on these social
+sites, this gives you an edge in finding new content to post to them!
+
+To get a head start, here are two exported data files of the tech-focused and general feeds I follow. You should be
+able to import them into your feed reader of choice — hopefully you find
+something you like! Note though that I don’t necessarily endorse any
+of these sites or authors; they may just have published one good thing years
+ago and I have no idea what they’ve been up to since.
+
+Ultimately, why should I bother with this?
+
+Like I said at the start, for me it comes down to three reasons:
+
+Firstly, it just works really well — better than alternative options. By
+that I simply mean that a feed reader really is the best way to keep up with
+blogs and posts! A favourite part of my day is having my morning coffee and
+catching up on feeds. I can pick them up when I have a few minutes to spare,
+and what I’ve read or not read is automatically synced between my devices.
+Easy, pleasant, simple, convenient, and I’m in control of what I read.
+
+The second reason is maybe not universally accepted by everyone, but in my
+opinion, it’s a healthier alternative to social media. I feel those platforms
+should be used only lightly and warily, if at all. Their very structure
+inherently promotes controversy and pile-ons rather than discussion; and fake
+“influencer” distortions over reality. While they may be a reasonable way to
+keep up with a small group of friends and family, opening the door to
+strangers being algorithmically boosted into your timeline, and the associated
+advertising and psychological manipulation (whether intentional or merely a
+side-effect) doesn’t feel healthy to me. Following blogs directly is a way to
+minimize or outright bypass these platforms and still benefit from the wide
+range of content out there on the web.
+
+And finally, the third reason is that I hope that directly following blogs
+and creators promotes the older, more independent, non-commercial version of
+what the Web could be. Rather than merely a more interactive cable TV channel
+controlled by big platforms, we can encourage anyone who has an interesting
+story to tell to share it online, and they can build up a dedicated audience
+independently, without having to pander and clickbait in the hope of
+algorithmically going viral. I want to support that version of the
+Web by choosing what to read, following small blogs, and enjoying reading long
+blog posts written simply because the author had something to say.
+
+Appendix: Why all the inconsistent names?
+
+An unfortunate complication when evangelizing or adopting feeds is the
+proliferation of opaque names that all mean essentially the same thing. Why is
+it like this?
+
+The concept of what we’re trying to publish is a “feed”. The
+original standardized format for publishing feeds was “RSS”: “Really
+Simple Syndication”. Several years later there was a schism in the community
+and a separate group produced a new standard format called “Atom”. Today, the
+choice of format is largely irrelevant, and essentially all feed readers
+support both. However, the name “RSS” caught on reasonably well, and
+so sometimes people will call the generic concept “RSS feeds” — confusingly
+even when the actual format in use might be Atom and not RSS!
+
+However, ultimately, none of these details matter in the slightest to
+users. The only reason to know that RSS, Atom, news feed, and web feed are all
+possible names is so you can find the link to the feed to plonk it into your
+feed reader, which will happily work with any format or name.
+


New Recommended Reading: People and Project Management
Sat, 30 Nov 2019 11:27:59 -0500
I’ve updated my 
+recommended reading list for software professionals (introduced
+and explained here) with a new section on people and project management.
+
+
+Don’t think you need the word “manager” in your job title to benefit from
+learning these skills. If you want to make yourself more useful to a team,
+teach others, help your projects succeed, or guide groups to making good
+decisions, these books will help.
+
+The 
+new section of the list is here.
+


More Recommended Reading: Information, Design, and UX
Fri, 24 May 2019 16:25:13 -0400
I’ve just added a new section to my 
+recommended reading list for software professionals (introduced
+and explained here). This new section covers the presentation
+of information, design, and user interfaces/experiences.
+
+
I’m not a designer, and reading these books won’t make you one either. But
+every developer should know how to present information and interfaces clearly
+and comprehensibly.
+
+You can find 
+the list itself here.
+


Recommended Reading List for Software Professionals
Sun, 19 May 2019 14:10:27 -0400
I’ve started putting together a list of the “core” books I recommend for
+people interested in exploring different facets of our field. I certainly
+don’t think you need to read all of them to be a capable software
+professional; rather it is the list I would put together if asked about how to
+learn more about specific areas.

+
+
This list came about because after compiling similar lists two or three
+times over the years in various places and formats, I’m following good
+development practice and factoring it out for reuse and sharing.
+
+You can find 
+the list itself here.
+
+The list is far from complete; in fact today I’m starting with just one
+area: the culture and history of our field. Over time I will add sections on
+technical system design and architecture; project and people management; data
+visualization, information presentation, and user experience design; etc. Let
+me know if there are areas you want covered, or if you have recommendations of
+your own!
+


Favourite Firefox Feature: Keyword Search Bookmarks
Thu, 28 Mar 2019 21:55:46 -0400
One of my favourite features of the Firefox web browser is surprisingly
+unknown and sadly underused. Keyword search bookmarks let you kick off a
+custom search of any site straight from the address bar (a.k.a. the “awesome
+bar”). For example, if you typed “w hedgehog”, you could go to
+the Wikipedia page for the little critters, or “img hedgehog”
+could show you a cute image search.
+This is similar to search engine DuckDuckGo’s “!bang commands”, except you
+can define your own new ones and pick whatever keyword you like, instead of
+being limited to pre-defined ones.
+
+You can set up a new search by right-clicking on any search field (try the one in the top-right on Wikipedia) and
+selecting the context menu option “Add a Keyword for this Search…”. In the
+dialog that pops up, just pick the shortcut you’d like to use (for example,
+for Wikipedia I use “w”).
+
+
+  
+  
+
+
+Alternatively, you can do the same thing by creating a regular bookmark
+through the full bookmark manager (not the sidebar) and filling in the
+“Keyword” field. In the bookmark URL, put the string “%s”
+wherever you want your search term to appear, and when you trigger the
+bookmark search via the keyword, it will be replaced with your query.
+
+
Some of my keyword searches that I use frequently:
+
+
+  w → Wikipedia
+  map → Google Maps
+  img → Google Images
+  yt → YouTube
+  py → Python docs
+  rs → Rust docs
+  mdn → Mozilla Developer Network
+
+
+It’s just super handy to be able to type “map 123 main st” or
+“w echidna” and get to the right place — especially on a phone.
+You can also make great use of this at work by setting up search bookmarks for
+your tools; like GitHub, JIRA, internal docs, etc.
+
+I believe Chrome can also do something similar, but it may involve a few
+more steps and configuring a custom search engine — I’m not sure.
+
+I hope this feature will delight you as much as it does me!
+


Hello 2023, Hello JSON Schema Community
Tue, 10 Jan 2023 00:00:00 GMT
Since Ben Hutton joined Postman to work full-time on JSON Schema the past year, other top contributors like Jason De
+


Introducing: Bowtie
Wed, 16 Nov 2022 00:00:00 GMT
A new tool for executing JSON Schema implementations & a call to help improve it


How the W3C Web of Things brings JSON Schema to the Internet of Things
Tue, 01 Nov 2022 06:00:00 GMT
Using JSON Schema at the W3C Web of Things to create an interoperability layer so that different IoT platforms, protocols and standards can operate together


Towards a stable JSON Schema
Fri, 21 Oct 2022 00:00:00 GMT
About this time last year, I hosted a discussion at the API Specification
+Conference about the future of JSON Schema. The most popular topic of discussion
+was, when JSON Schema is going to be "done".
+


Using Dynamic References to Support Generic Types
Wed, 31 Aug 2022 00:00:00 GMT
A step in the right direction for modelling data with JSON Schema


Fixing JSON Schema Output
Sat, 23 Jul 2022 00:00:00 GMT
I have a problem: when I read GitHub issues, they occasionally resonate with me, and I obsess about them until they're resolved.  That may not sound like a problem to some, but when that resolution c
+


The Architectural Scope of JSON Schema
Fri, 15 Jul 2022 00:00:00 GMT
What actually is JSON Schema?!


Joining Postman
Sun, 03 Jul 2022 00:00:00 GMT
In the last couple of weeks, you may have seen that Postman has been investing
+heavily in JSON Schema's future by hiring some of its top contributors giving
+them the opportunity to work full time on 
+


And Then There Were Three
Thu, 23 Jun 2022 00:00:00 GMT
More full-time contributors is better, right?


Hello World, Hello Postman
Thu, 16 Jun 2022 00:00:00 GMT
Oh hello there! A second person is now working full-time on JSON Schema.


Astonishing Serializations & Schemas of Hyperborea
Wed, 18 May 2022 00:00:00 GMT
Using JSON Schema for validating role-playing character sheets.


Get started with JSON Schema in Node.js
Mon, 16 May 2022 06:00:00 GMT
Learn how to use JSON Schema for validation in your Node.js applications.


How Tyler Technologies reduced its client feedback loop with JSON Schema
Mon, 09 May 2022 06:00:00 GMT
Using JSON Schema at Tyler Technologies meant showing added value to clients could take minutes rather than days or in some cases weeks.


It all starts with applicability - JSON Schema Fundamentals part 1
Mon, 21 Mar 2022 00:00:00 GMT
We explore the fundamental JSON Schema concepts: Applicability, Subschemas, and Assertion Boolean Logic - Everyone needs good fundamentals.


JSON Schema joins the OpenJS Foundation and 2022 updates
Fri, 11 Feb 2022 00:00:00 GMT
Find out what we've been doing in 2021 and read about our biggest development yet... joining the OpenJS Foundation. Oh, and WE'RE HIRING!


JSON Schema in 5 minutes
Fri, 21 Jan 2022 00:00:00 GMT
Everyone needs good fundamentals, and understanding the basics of JSON Schema changes how you read, reason, and develop Schemas.


クックパッド株式会社におけるJSON Schemaの活用事例
Mon, 17 Jan 2022 06:00:00 GMT
クックパッド株式会社ではJSON Schemaを活用することによって、クックパッドマートにおける販売者の商品登録時の正確性と体験を改善し、商品審査にかかる運営上の負担を大幅に軽減することができました。


JSON Schema deduplicated complex logic and validation at Cookpad
Thu, 09 Dec 2021 06:00:00 GMT
Using JSON Schema at Cookpad improved the accuracy and experience when registering products, and has greatly reduced the operational burden of product screening.


Validating OpenAPI and JSON Schema
Wed, 08 Dec 2021 00:00:00 GMT
Dynamic references make it possible to validate the schemas in an OpenAPI document even though OpenAPI does not constrain which JSON Schema dialects can be used.


Why JSON Schema needs a Code of Conduct
Wed, 24 Nov 2021 06:00:00 GMT
The community of JSON Schema is growing, and the community needs a culture for interactions to avoid past mistakes.


JSON Schema bundling finally formalised
Wed, 04 Aug 2021 06:00:00 GMT
Existing tooling developers have created their own approaches to bundling JSON Schema and OpenAPI documents, but that can lead to errors. Bundling is now standardised.

Brett.is

Thinking about transactional email

Managing Go dependencies with git-subtree

Keep it simple

Vendor dependencies

Maintain the full source code of each dependency in each repository

In come git-subtree

Adding a new dependency

Updating an existing dependency

Using tags/branches/commits

Making it all easier

Why not…

Godep/<package manager here>

git-submodule

Something else?

Write code every day

Forge configuration parser

Project overview

How it works

Conclusion

What I'm up to these days

Javascript Documentation Generation

Python Redis Queue Workers

worker.py

qw

Lets Make a Metrics Beacon

Tags

Metrics

Syncing

How To Do It

Aggregating Data

Syncing Data

Unload

Conclusion

TODO

Links

Goodbye Grunt, Hello Tend

Installation

Usage

Example CLI Usage

Config File

With Make

Conclusion

Sharing Data from PHP to JavaScript

page.php

page.php

page-script.js

Cookieless User Tracking

Browser Fingerprinting

Local Storage

ETag Header

Example Server

Redirect Caching

Example Server

Ever Cookie

Other Methods

References

My New Website

The Fastest Python JSON Library

The Battle of the Caches

Browser Fingerprinting

What is Browser Fingerprinting?

What can it used for?

How do we do it?

Some Code

Your fingerprintjs Fingerprint: Could not generate fingerprint

Making Citizen’s Issues Accessible

A Better Tech-a-thon: ground-breaking collaboration on Public Interest Technology

Navigating Career Changes with Chocolate Milk Diplomacy

ABOUT THE COURSE CREATORS

COURSE SUMMARY

MORE RESOURCES

Building a Database to End School Violence

Seeing the forest through the trees

An idea born of tragedy

The Athena Project

Current state and next steps

Get Involved

Volunteers make the 2021 Summer Create-a-thon a Success

Who participated