Mixu's blog

Curl cheatsheet

Mixu — Tue, 29 Jan 2013 08:00:00 GMT

Curl is an awesome tool. I keep forgetting the switches though, so here's a cheatsheet.

Simple GET request

curl -k "https://localhost/foo?bar=baz&abc=def"

JSON POST or PUT request

curl -k -H "Content-Type: application/json" -X POST -d '{"accountName":"test","value":"hello"}' https://localhost/foo

curl -X "PUT"

... for a PUT

POST a file

curl ... --data-binary @filename

Fake a /etc/hosts entry and a Host: header with curl

curl -vvv --resolve 'book.mixu.net:80:123.145.167.189' http://book.mixu.net/

Make a request with basic auth enabled

curl -vvv -u name@foo.com:password http://www.example.com

or:

curl --user name:password http://www.example.com

Set the Referer header

curl -e http://curl.haxx.se daniel.haxx.se

Set the User Agent header

curl -A "Mozilla/4.73"

curl --user-agent "Mozilla".

Set Cookies

curl -b "name=Daniel"

curl --cookie "name=Daniel"

Time a request (connect time + time to first byte + total tile)

curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://google.com

Downloading files from Github

curl -O  https://raw.github.com/username/reponame/master/filename

Restaurants in SF

Mixu — Thu, 20 Dec 2012 08:00:00 GMT

I’ve been living in SF for quite a while now, so I figure I should list some of the restaurants that I like (the top ~45 to be exact). These are mostly places where you can get food quickly for lunch/dinner and that are walking distance from where I live.

Coffee

Philz Coffee. The best coffee in SF (in my opinion); the one closest to the center is on Van Ness (good if you go to the Tenderloin for lunch). The other Philz in the city center is at the Caltrain in SOMA (all locations). I always stop by when I’m close. Protip: don’t get the coffee with sugar, because they put too much and it detracts from the taste. Tantalizing Turkish, Aromatic Arabic and Soo Good (light roast) are my favorites.

Machine Coffee at 6th/Golden Gate is also quite good as are their sandwitches.

Blue Bottle. I would describe the coffee as artsy (as in “this coffee thinks it’s better than me”) – good for the occasional cup but I’m not a huge fan.

We have Peet’s coffee beans at the office so I haven’t really tried Peet’s at a store. In any case, these are all better than Starbucks, which is utter crap unless you get one of the specialty blends of the month rather than their regular coffees.

Japanese noodles

There are basically two good noodle places in the city center:

Ramen Underground (Union Square towards Financial District). RU is a short walk from Union Square. Their specialty is the spicy miso ramen. I recommend getting that with extra noodles and BBQ pork. Open on Saturdays as well.
Katana-ya (between Union Square towards Tenderloin). More choices than RU, including cold ramen and the toppings are better.

+Expect to wait ~15 minutes. Suzu in Japantown is also good, on the other side of the walkbridge, downstairs. Once there, go to Moyo’s for some frozen yogurt.

If you have a car, drive to San Mateo, as there are some delicious noodle places there:

Santa Ramen (in San Mateo). Two words: premium pork! Mind-blowingly delicious.
Ramen Dojo (in San Mateo). Also very good. Prepare to wait in line though.

+If you don’t have a car, get a Zipcar and drive. Seriously, it’s worth it.

Japanese curry

Muracci’s (Union Square towards Financial District) specializes in Japanese curry. I go there so often it’s ridiculous, only open till 6pm and only on weekdays though.

Kare-Ken (Tenderloin) is open after 5:30 till fairly late, however, their portions are smaller and the curry sauce isn’t quite as delicious as at Muracci’s.

Vietnamese sandwitches (banh mi)

These are a combination of Vietnamese -style veggies and meat in sandwitch form. Tasty and cheap.

Saigon Sandwitch (Tenderloin). The most famous banh mi place, this is for the banh mi snob. Their sandwitches are a bit spicier than the others and a bit smaller, but also taste slightly better.
Lee’s (Tenderloin). I like Lee’s as well, it’s across from Saigon. Their sandwitches are a bit bigger and they have more varieties, but the bread is less tasty.
Hoang Dat (Tenderloin). A smaller shop that’s up the street about two blocks. Nice if you don’t want to walk through the Tenderloin. Specialty is a sandine banh mi.

+Outside Tenderloin, there are two other places I’d recommend:

Little Paris (Chinatown) is the place to go if you want Vietnamese sandwitches in Chinatown. I love this place and I was happy to find that they reopened (they moved about six months ago).
Fresh Brew Coffee (Union Square) is a small coffee shop closer to Union Square where you can get banh mi with avocado in it. It is expensive compared to the Tenderloin or Chinatown, but bacon and avocado are a winning combination.

Other kinds of sandwitches

HRD Coffee Shop (SOMA) is awesome. They have really great sandwitches, and their specialty are the sandwitches with Korean spices. I miss living in SOMA.
King Foot Subs (Union Square; Bush and Powell) offers similar sandwitches, though they aren’t quite as good as HRD. They are open late, however.
Molinari’s (North Beach). Prosciutto sandwitches, need I say more?
Morty’s Delicatessen (Tenderloin). Really delicious deli sandwitches.
Miller’s East Coast Deli and Freddie’s Sandwiches are near each other, I like Freddie’s a lot and Miller’s has a good pastrami sandwitch.
Cafe Venue. Next to the Westfield Mall, really big salads during lunchtime, usually enough for two meals.

+Chinese dumplings (dim sum)

There are hole-in-the-wall type places where you can get two meals worth of food for 6-8 bucks:

Good Mong Kok Bakery. This is where all the Chinatown locals queue up, and it is good.
Delicious Dim Sum. More accessible than Good Mong Kok, also good, very close by to Good Mong Kok.

+Chinatown protip: there are small side streets that mostly avoid the crowds, use them if you don’t feel like dodging people.

Vietnamese

Pho = thin noodles with a ligher soup and veggies. The vermicelli noodle salad with spring rolls is also delicious. Check out:

Pho 2000 (Tenderloin): big portions, good stuff.
Pho Tan Hoa (Tenderloin/Union Sq): closer to Union Square, fast service, big portions of pho.
Miss Saigon (~6th). Sketchy neighborhood but the food is good.
Tu Lan (~6th). Delicious, but closed down until ~2013. Tu Lan is legendary, but it was closed for repeated health code violations so I’m not sure I’ll go there very often when they reopen.
Loving Hut (Westfield Mall but also Chinatown). One location is in the Westfield Mall and I’d guess this is healthy food. Fully vegan, but I couldn’t tell. I always get the vermicelli noodle salad with spring rolls.

My Debian installation tips

Mixu — Wed, 24 Oct 2012 07:00:00 GMT

So, after some time with Arch Linux, I've moved to Debian testing. Debian testing seems to strike a nice balance between being new and not breaking things (Debian stable on the other hand has really old packages). Compared to Arch, upgrades have thus far been a lot less painful. Arch, while lovely, has a tendency to break things and I was always rather annoyed with python being python3 rather than python2 like the rest of the world expected it to be. I miss the minimal base install from Arch though.

First steps

I used the amd64 (e.g. x86-64) weekly testing build rather than the stable build, because, well it gets me on testing without needing to do a stable install first and comes with more up-to-date packages.

While none of the docs make it clear, all you need is the first CD (or the netinstall build) to start the install.

Edit sudoers

su -c "visudo"

At the end of the file, add:

yourusername ALL=(ALL) NOPASSWD: ALL

Copy ssh keys over from old machine

Update and install packages

Add "contrib" and "non-free" after "main" to /etc/apt/sources.list - this is needed for stuff like the flash plugin.

If there are lines mentioning "cdrom:" as a source, delete or comment out those lines.

sudo apt-get update && sudo apt-get upgrade
+
+sudo apt-get install git mlocate mercurial nitrogen wicd-gtk \
+gnome-icon-theme htop unzip openssl chromium-browser zsh \
+xterm epdfview mysql-server ruby tmux curl wget redis-server \
+xcursor-themes xarchiver gzip bzip2 zip unzip unrar p7zip \
+meld mplayer make gcc libc-dev build-essential openssl \
+libssl-dev libx11-dev libxinerama-dev guake xfce4-notifyd \
+keychain flashplugin-nonfree lxappearance tree

Install node

mkdir other
+cd other
+wget http://nodejs.org/dist/v0.8.12/node-v0.8.12.tar.gz
+tar xvvf node-v0.8.12.tar.gz
+cd node-v0.8.12
+./configure && make && sudo make install

Install nwm, my window manager

git clone git://github.com/mixu/nwm.git
+cd nwm
+npm install

Create ~/nwm.sh and chmod +x:

#!/bin/sh
+/usr/local/bin/node /home/m/nwm/nwm-user-sample.js 2> ~/nwm.err.log 1> ~/nwm.log

Create /usr/share/xsessions/nwm.desktop:

[Desktop Entry]
+Encoding=UTF-8
+Name=nwm
+Comment=This session starts nwm
+Exec=/home/m/nwm.sh
+Type=Application

Set the user shell from bash to zsh

sudo chsh

Set the default xdg-open action for PDFs

In case you don't know what xdg-open is - it's the thing that decides what happens when you try to open a file (e.g. a downloaded file from Chrome).

I don't want to open PDFs and PS files in Gimp, I want to open them in a document viewer

xdg-mime default evince.desktop application/pdf

xdg-mime default evince.desktop application/postscript

To query the default app: xdg-mime query default $(xdg-mime query filetype FILENAME)

Configuring appearance

Download https://wiki.ubuntu.com/Artwork/Incoming/DustTheme and unzip the theme to ~/.themes/

I find the easiest way to configure gnome2 is to use lxappearance, since it allows you to set the gnome theme and x11 cursors in a few clicks.

Guake: disable smart terminal names

gconftool-2 --set /apps/guake/general/use_vte_titles --type boolean false

Configure git

Edit ~/.gitconfig:

[color]
+ui = true
+[user]
+email =
+name =
+[core]
+editor = nano
+[alias]
+lg = log --graph --pretty=format:'%Cred%h%Creset-%C(yellow)%d%Creset %s %Cgreen(%cr)%C(bold blue)%Creset' --abbrev-commit --date=relative
+[push]
+default = current

Configure zsh

# Lines configured by zsh-newuser-install
+HISTFILE=~/.histfile
+HISTSIZE=1000
+SAVEHIST=9999
+unsetopt beep
+bindkey -e
+# End of lines configured by zsh-newuser-install
+# The following lines were added by compinstall
+zstyle :compinstall filename '/home/m/.zshrc'
+
+autoload -Uz compinit
+compinit
+# End of lines added by compinstall
+
+bindkey "\e[1~" beginning-of-line # Home
+bindkey "\e[4~" end-of-line # End
+bindkey "\e[5~" beginning-of-history # PageUp
+bindkey "\e[6~" end-of-history # PageDown
+bindkey "\e[2~" quoted-insert # Ins
+bindkey "\e[3~" delete-char # Del
+bindkey "\e[5C" forward-word
+bindkey "\eOc" emacs-forward-word
+bindkey "\e[5D" backward-word
+bindkey "\eOd" emacs-backward-word
+bindkey "\e\e[C" forward-word
+bindkey "\e\e[D" backward-word
+bindkey "\e[Z" reverse-menu-complete # Shift+Tab
+# for rxvt
+bindkey "\e[7~" beginning-of-line # Home
+bindkey "\e[8~" end-of-line # End
+# for non RH/Debian xterm
+bindkey "\eOH" beginning-of-line
+bindkey "\eOF" end-of-line
+# for freebsd console
+bindkey "\e[H" beginning-of-line
+bindkey "\e[F" end-of-line
+# for guake
+bindkey "\eOF" end-of-line
+bindkey "\eOH" beginning-of-line
+bindkey "^[[1;5D" backward-word
+bindkey "^[[1;5C" forward-word
+bindkey "\e[3~" delete-char # Del
+
+alias ls='ls --color=auto -F'
+alias ga='git add'
+alias gp='git push'
+alias gl='git log'
+alias gs='git status'
+alias gd='git diff'
+alias gdc='git diff --cached'
+
+autoload -U colors && colors
+
+PROMPT="%{$fg_bold[white]%}[%1~]%{$reset_color%} "
+
+PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting
+[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"
+
+gr() {
+   local cmd="$"
+   for dir in /home/{FILL IN REPOSITORIES HERE}; do
+    (
+     print "\nin $dir"
+     cd $dir
+     eval "$cmd"
+    )
+   done
+}
+
+alias grs="gr git --no-pager status -sb"
+alias grl="gr git --no-pager log --decorate --graph --oneline -n 3"
+alias grd="gr git diff"
+alias grdc="gr git diff --cached"
+alias grn="gr npm ls"
+
+eval `keychain --eval --agents ssh id_rsa`
+
+# Stuff for git
+parse_git_branch () {
+    git branch 2> /dev/null | grep "" | sed -e 's/ (.)/ @\1/g'
+}
+
+# colors for prompt
+BLACK=$'\033[0m'
+RED=$'\033[38;5;167m'
+GREEN=$'\033[38;5;71m'
+BLUE=$'\033[38;5;111m'
+YELLOW=$'\033[38;5;228m'
+ORANGE=$'\033[38;5;173m'
+
+function precmd() {
+    export PROMPT="%{$fg_bold[white]%}[%1~%{$reset_color%}%{$YELLOW%}$(parse_git_branch)%{$fg_bold[white]%}]%{$BLACK%}%\ "
+}

Writing about technical topics like it's 2012

Mixu — Thu, 26 Jul 2012 07:00:00 GMT

For some reason, a lot of technical writing is still done as if the reader didn't have access to the Internet. Having written two books (Mixu's Node book and Single page apps in depth) in the past year, I've had some time to think about what good technical writing might be. There are several things that could be improved:

1. Assume the reader has Internet and Google capabilities. Skip the parts that are covered better elsewhere or that will go out of date quickly. For example, installation tutorials are never up to date. I'll generally skip the sales pitch for the technology as well, since I'm here to tell you about the tech, not sell you on using it.

2. Keep it short and to the point. I hate books that try to be cute or clever with me - if I want a story, I'll go read fiction. Similarly, don't surprise the reader with irrelevant sections. Just because you wrote it doesn't mean it should exist. I keep a crap file where all the useless text goes, and I usually have a couple of chapters worth of text that I delete. I've written several chapters on the history of a piece of tech, and always deleted them in the end.

3. Use teasers, lists, tables and code blocks. Assume that the reader can get bored quickly, so throw in something interesting quick (a teaser) and support skimming readers with subheadings, lists and other structural elements.

4. Don't treat everything as equally important. The easiest way to write a book about a piece of tech is to take the API, and then cover each and every function in it. It's also the worst possible way to do it. Unless you are writing the API docs, you should avoid talking about everything and focus on the interesting parts.

As a reader, if I want to find out more, I can always read the API or code. What is much more important is understanding what are the neat things that I can do with the new tech, what kinds of patterns are common in using it and why you as the author think they are good or bad. Talk about how the pieces fit together. If everything is equally important, then nothing stands out. I like getting the author's angle on things, even if I might disagree with their choices, I still feel like I gain insight.

5. Give me a choice in formats. Provide HTML, PDF, epub and mobi versions of long texts. Converting html for the Kindle just requires running one command with Calibre, so I wish more authors gave me the option to download their writing. For example, I wish I could get Google's tutorials for Android as a Kindle mobi book, since that's how I like to read long form writing.

6. Help me navigate. Often, when I like what I read, I'll want to read more. Don't force me to go page by page through your writing. If you have a blog, have an archive that can show all the content, and let me get an idea of what else you've written about at a glance (a most popular list always helps).

More about writing:

http://www.paulgraham.com/writing44.html

http://bighow.com/news/the-art-of-great-writing-60-writing-tips-from-6-alltime-great-writers

http://dannystack.blogspot.com/2009/01/joss-whedons-top-10-writing-tips.html

Git tips and tricks

Mixu — Fri, 06 Apr 2012 07:00:00 GMT

This post is basically a collection of things I think I should remember about git: setup, commands, multiple repos, more obscure options etc.

Configuration

Setting up short aliases for you git commands changes everything.

Two things: enable colors, and make git push push the current branch to a branch of the same name (and not all the branches, which is the default). This means when you do git push, it pushes your working branch but not other branches.

git config --global color.ui true
+git config --global push.default current

Maybe worth it:

git config branch.autosetuprebase always
+git difftool -t 
+git mergetool -t

Zsh/bash aliases

These are small things, but they make things so easy. I instinctively type gs after changing directory nowadays.

alias ga='git add'
+alias gp='git push'
+alias gl='git log'
+alias gs='git status'
+alias gd='git diff'
+alias gdc='git diff --cached'
+alias g='git'
+git config --global alias.lg "log --graph 
+  --pretty=format:'%Cred%h%Creset-%C(yellow)%d%Creset %s %Cgreen(%cr)%C(bold blue)<%an>%Creset'
+  --abbrev-commit --date=relative"

Apparently, the "git lg" above can be replicated fairly well with:

git log --decorate --graph --oneline

Zsh shell prompt

This one is way simpler than those fancy ones offered elsewhere, but it works well for me:

# get the current branch
+parse_git_branch () {
+    git branch 2> /dev/null | grep "" | sed -e 's/ (.*)/ @\1/g'
+}
+
+# colors
+BLACK=$'\033[0m'
+RED=$'\033[38;5;167m'
+GREEN=$'\033[38;5;71m'
+BLUE=$'\033[38;5;111m'
+YELLOW=$'\033[38;5;228m'
+ORANGE=$'\033[38;5;173m'
+
+function precmd() {
+    export PROMPT="%{$fg_bold[white]%}[%1~%{$reset_color%}%{$YELLOW%}$(parse_git_branch)%{$fg_bold[white]%}]%{$BLACK%}%\ "
+}

Managing multiple git repositories

A fairly simple way of doing this is to add a function that you can use to run the same command in multiple directories. This one is for zsh:

gr() {
+  local cmd="$*"
+  for dir in /home/m/repos/{one,two,three}; do
+   (
+    print "nin $dir"
+    cd $dir
+    eval "$cmd"
+   )
+   done
+}
+
+alias grs="gr git --no-pager status -sb"
+alias grl="gr git --no-pager log --decorate --graph --oneline -n 3"
+alias grd="gr git diff"
+alias grdc="gr git diff --cached"
+alias grn="gr npm ls"

In the example below, I 1) switch all working repos to master, 2) fetch the recent remote changes, 3) check the status of every repository, and 4) after going into those couple of dirs and stashing stuff (not shown in detail), pull all repos to update them

gr git checkout master
+gr git fetch
+grs
+in /home/m/repos/one
+## master
+in /home/m/repos/two
+## master...origin/master [behind 8]
+in /home/m/repos/three
+## master...origin/master [behind 4]
+ M Makefile
+gr git pull

Add all changes or all changes to existing files

$ git add -A

Adds all files and file deletes, so you don't need to enter them manually.

$ git add -u

Same, but doesn't add files that didn't exist before.

Log-fu

$ git log -p

.. to show the patches alongside the log

git log --author="git config --get user.name" --pretty=format:'%h %an %Cred%ar %Cgreen%s' v1.0.1..master

git supports friendly dates specs:

git log HEAD@{one.day.ago}

Shows the commits that you've made between "refactor" and "master", e.g. for emailing a changelog with your commits:

6591466 Mikito Takada 7 months ago Port over more code
+1f464fb Mikito Takada 7 months ago Refactor window
+41a3ec9 Mikito Takada 9 months ago Handle switching from window to root window

$ git log staging -1 -p -- my/subdir

Show the latest commit that affects the my/subdir path on the branch "staging" with the patch.

$ git log --stat

Show the changed files only.

$ git log --author=foo

Only commits by author foo.

$ git log --stat -- subdirectory

Show the commits affecting only "./subdirectory", with a summary of the files (--stat).

For example:

$ git log --stat -- .

Shows the summary of commits affecting the current directory.

$ git log --grep='.foo.'

Grep for a specific log message.

$ git log --since="1 week ago" --until="2 weeks ago"

Filter commits by date.

$ git shortlog -sn

Shows a "top contributors list" by commits.

Selecting things

Full SHA1
Partial SHA1 - at least 4 characters and unique
branch^ first parent
branch^2 merge parent
branch~5 five first-parents back
branch:path/to/file directly get a blob
branch@{yesterday} local date based reference
branch@{5} the 5th last value of branch (locally)

Diff-fu

Differences between the staged files and the last commit:

$ git diff --cached

I use that a lot just before committing the changes to check what my commit changed.

So git diff is for the files that are unstaged, while git diff --cached is for staged files.

What's different between my branch and some other branch, e.g. master?

$ git diff my_branch..master

What changed in the last three commits?

$ git diff HEAD^3

Ignore whitespace?

$ git diff HEAD^ --word-diff

What's different between my branch and some other branch in subdirectory (or file)?

$ git diff my_branch..master subdirectory

Remember, git diff outputs an applicable patch:

$ git diff > my.patch
+$ git apply < my.patch

For example, I once used git diff to generate a patch for a folder that was missing it's .git directory, and then applied that patch to a live environment (patching a test VM):

git diff --no-index --diff-filter=MRC -- ./lib ../myapp/lib > my.patch
+cd ../myapp/
+git apply ../myapp-git/my.patch

Grep-fu

$ git grep -e foo --or -e bar --not -e baz

Grep for foo or bar, but exclude the expression baz.

$ git grep --untracked foo

Include untracked files in grep.

$ git grep --cached foo

Include cached (staged) files in grep.

$ git grep --no-index foo /var/bar

Use "git grep" instead of regular grep, grep for foo in /var/bar. Compared to regular grep, git grep is nicer due to the colors and paging defaults.

List all require('foo') statements content (sed to filter out just the string).

git grep -h -e "require[(]'[a-zA-Z]" | sed -E "s/.require(.([a-zA-Z][^']+).)./\1/" | sort | uniq

Creating a feature branch and merging it back

Checkout the branch you want to base your work on:

$ git checkout master

Then create a branch and check it out:

$ git checkout -b local_name

Push the branch to the remote

$ git push -u origin local_name

Later on, merge it:

$ git checkout master

$ git merge local_name

Look at the unmerged branches:

$ git branch --no-merged

Look at branches that contain a particular commit (e.g. when cherrypicking):

$ git branch --contains 1234abcd

Git hooks

Installing:

cp pre-commit.sh .git/hooks/pre-commit
+chmod a+x .git/hooks/pre-commit

Examples:

deploying a static website
checking commits and doing cleanup
integration with external systems
email notifications

Deploying over git, with a post-update checkout hook

Git works over ssh

If all you need is a place for storage, you just need sshd and disk space.

Setting up a remote repository

Make local repo, commit stuff, etc.
ssh to the server: GIT_DIR=/path/to/repo git init --shared
Next, tell the local repo about the server: git remote add origin git+ssh://hostname/path/to/repo
Push to the server from the local repo: git push origin master
Clean up the local repo so that you can pull from the remote server: git config branch.master.remote origin git config branch.master.merge refs/heads/master

Excluding files

~/.gitignore for globally excluded files:

npm-debug.log
+.DS_Store
+.DS_Store?
+.Spotlight-V100
+.Trashes
+Icon?
+ehthumbs.db
+Thumbs.db

.git/info/exclude for files you want to exclude without changing the gitignore file:

conf/*

.gitignore for a checked in list of excluded paths:

node_modules/

Whitelisting a directory (note: needs the rest of the path):

static/mixu.net/blog/*
+!static/mixu.net/blog/assets/

Stashing changes

$ git stash
+$ git stash list
+$ git stash apply

You can specify a specific stash item from the git stash list output to apply or remove:

$ git stash apply stash@{0}
+$ git stash drop stash@{0}

Blaming

$ git blame path/to/file

See who changed what.

$ git blame abce1234^ path/to/file

See the blame, starting one commit before abce1234. E.g. when you need to trace back a particular line of code in history.

Working with remotes

Changing from Github to Bitbucket

$ git remote -v # look at remotes

$ git remove rm origin # delete

$ git remote add origin git://addr/to/repo # add new remote

$ git push -u origin master # push to new remote

Adding a upstream repo to pull in new changes from a forked project

$ git remote add upstream git://addr/to/repo

$ git checkout master && git pull upstream master

Keeping the number of merges lower on small commits

For small commits made on top of a frequently changing branch like master, you might want to rebase your local changes on top of the current remote before you merge a feature branch (more in depth):

$ git pull --rebase

$ git push

Branching

List all branches:

$ git branch -a

Unmerged branches:

$ git branch --no-merged

Checkout a branch (protip, zsh does git branch completion):

$ git checkout name

If there is a branch called origin/name and you checkout name, then name will be set to track origin/name automatically (that means that git pull works, for example).

Create a feature branch based on the current branch:

$ git checkout -b name

Create a orphan branch (no history) for storing documentation or resources:

$ git checkout --orphan

Merge branches:

$ git merge

Apply a commit from another branch:

$ git cherry-pick sha1_of_commit

Reset the branch to the current head:

$ git reset --hard HEAD

When a branch tracking a remote has become outdated (e.g. you are on staging now but your commits have diverged):

$ git reset --hard origin/staging

Reset a single file to HEAD:

$ git checkout -- path/to/file

Reverting (unapplying) a bad commit:

$ git revert sha1_of_commit

Taking the current branch and creating a version that's squashed into one commit:

$ git checkout -b new-branch
+$ git rebase -i

During the rebase, only pick one commit and squash the rest.

Total number of remote branches

First, remove local branches that do not exist on origin (--dry-run if you want to preview):

$ git remote prune origin

$ git branch -r | wc -l
+ 123

Get the latest commit of all remote branches

[gist id=2313880]

Get the latest commit of all remote branches, and summary, ordered by age

Use this Node script:

https://gist.github.com/91ba46dee32b221b3a84

Delete a local or remote branch

Delete a local branch:

git branch -d some_branch_name

Really delete a local branch, even if git complains about it in the previous command:

git branch -D some_branch_name

Delete a remote branch, e.g. in Github:

git push origin :some_branch_name

The reason for that syntax is that you can do git push origin local_branch_name:remote_branch_name so what that line above is doing is essentially pushing NULL to the remote branch, thereby killing it off.

In git >= v1.7.0:

git push origin --delete

Things I should still try out

git add -i
+git reflog

Mechanical keyboards rock!

Mixu — Fri, 17 Feb 2012 08:00:00 GMT

I'm a big believer in having the best tools possible for the job. I've gone through at least 3 Microsoft Ergonomic Keyboard 4000's, and used to think those were the best keyboards ever (as did Jeff Atwood).

Well, there is whole world of keyboards that are even better, namely: buckling spring keyboards (e.g. the IBM Model M), Cherry MX switch keyboards and Topre switch keyboards.

I got this one for my place - Cherry MX brown switches, which are less noisy (though definitely they could be even more silent):

The major difference between a mechanical keyboard and a regular one is in the feel of the keys. Mechanical keyboards have metallic springs, while cheaper keyboards just have some flexible plastic. After a short while typing on the MS Ergonomic keyboard started feeling like typing on a mushy piece of plastic compared to the mechanical keyboard. I do kind of miss the more ergonomic split layout, but typing is just a lot nicer overall on a mechanical keyboard.

I ended up buying two keyboards, both with Cherry MX switches (~110 USD from here). I didn't get a buckling spring keyboard, because they are more noisy and there aren't any space-saving tenkeyless models available. The third alternative - keyboards with Topre switches - are more expensive (~300+ USD) and have been described as less tactile. I'll probably get a Topre keyboard eventually anyway, just so I can try it out.

Here is the new keyboard I use at work, which has Cherry MX blue switches (a bit more noisy than my other keyboard, but definitely one I prefer):

Best purchase in a long time!

How to: thinkpad_acpi and fan control on Arch

Mixu — Thu, 19 Jan 2012 08:00:00 GMT

From the "this might help some random person" department - setting up manual Thinkpad fan control in Arch Linux.

Edit /etc/rc.conf. Add thinkpad_acpi to MODULES=() in the hardware section.
+
Create a new file enabling fan control as /etc/modprobe.d/thinkfan.conf:
+

options thinkpad_acpi fan_control=1

Reload the thinkpad_acpi module:

sudo modprobe -r thinkpad_acpi && sudo modprobe thinkpad_acpi

To view the current fan speed:

[~]$ cat /proc/acpi/ibm/fan
+status: enabled
+speed: 2887
+level: 4
+commands: level ( is 0-7, auto, disengaged, full-speed)
+commands: enable, disable
+commands: watchdog ( is 0 (off), 1-120 (seconds))

To control fan speed:

# echo level 0 | sudo tee /proc/acpi/ibm/fan (fan off)
+# echo level 2 | sudo tee /proc/acpi/ibm/fan (low speed)
+# echo level 4 | sudo tee /proc/acpi/ibm/fan (medium speed)
+# echo level 7 | sudo tee /proc/acpi/ibm/fan (maximum speed)
+# echo level auto | sudo tee /proc/acpi/ibm/fan (automatic - default)

For easy control from the command line:

function fan() {
+  sensors
+  echo level $@ | sudo tee /proc/acpi/ibm/fan
+}

Now you can run fan 2 to set the fan level to 2 and so on.

To view fan speed, install lm_sensors following the instructions on the Arch wiki. Example output:

[~]$ sensors
+acpitz-virtual-0
+Adapter: Virtual device
+temp1:        +37.0°C  (crit = +99.0°C)
+
+thinkpad-isa-0000
+Adapter: ISA adapter
+fan1:        2904 RPM
+
+coretemp-isa-0000
+Adapter: ISA adapter
+Physical id 0:  +39.0°C  (high = +86.0°C, crit = +100.0°C)
+Core 0:         +37.0°C  (high = +86.0°C, crit = +100.0°C)
+Core 1:         +37.0°C  (high = +86.0°C, crit = +100.0°C)
+Core 2:         +35.0°C  (high = +86.0°C, crit = +100.0°C)
+Core 3:         +39.0°C  (high = +86.0°C, crit = +100.0°C)

Comments

Mateusz Loskot: Thanks for the quick and clear how-to. +FYI, I used it in my response to one of thinkfan questions on the Arch forum: +https://bbs.archlinux.org/viewtopic.php?pid=1112691#p1112691

Performance benchmarking Socket.io 0.8.7, 0.7.11 and 0.6.17 and Node's native TCP

Mixu — Tue, 22 Nov 2011 08:00:00 GMT

I've been working with Socket.io quite a bit recently. It's a great library. However, after upgrading to 0.8.x, I ran into problems with increased CPU usage. Since performance is very important for high traffic pubsub implementations, I decided to investigate this further - and try to quantify the performance impact of upgrading to a newer version of Socket.io.

I wrote a benchmarking suite (siobench). The benchmark is rather simple. Clients connect one at a time, and a new client is only allowed to connect when the previous one is connected. When the server has used up 5000 milliseconds of CPU time, the benchmark is stopped. Every second, every connected client sends a single message which is echoed back by the server (more details).

This workload is geared towards a situation where Socket.io is used to notify people of things as part of a larger application: e.g. most of the load is assumed to be idling connections rather than real-time messaging like in, say, a multiplayer game.

The "end of test" condition is 5000 ms of CPU time, because this seemed to be a easy way to give all implementations the same amount of time. CPU usage % is not accurate, since it is dependent on how much CPU time the process gets over a particular amount of wallclock time. In the graphs the CPU usage % calculated over a 100ms interval, while usertime and systime are the actual numbers reported at that particular time.

Summary

+ + +

Node (0.4.12) using tcp	~ 8000 connections on a single core
socket.io 0.6.17 using websockets	~ 2300 connections on a single core
socket.io 0.7.11 using websockets	~ 1800 connections on a single core
socket.io 0.8.6 using websockets	~ 1900 connections on a single core

+Remember, this is just one server on one core, with 5000 ms of CPU time on that core. The rest of the cores are used to generate sufficient load. The full graphs are at the end of the post.

Note that the absolute numbers are mostly unimportant - I ran this on the following 15" Macbook Pro running Arch with the 3.1.04 Linux kernel in Virtualbox with 4096 Mb of RAM, a SSD and four cores (Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz GenuineIntel GNU/Linux). You can get numbers that are more representative of your system by getting siobench and running it:

Usage: node siobench.js [env]
+A tool for benchmarking your Socket.io server.
+
+Available environments:
+    0.6.17
+    0.6.17_poll
+    0.7.11
+    0.8.7
+    0.8.7_poll
+    tcp

You can also write your own benchmarks under ./bench, by writing a new server.js (example #1, #2) and a new client.js (example #1, #2). Each benchmark has it's own set of npm dependencies installed, so that one can run benchmarks against many versions of socket.io.

Some notes on performance

The relative performance is more interesting.

First, the node TCP speed represents the highest achievable performance on this benchmark, since it only uses the built-in TCP implementation. Compared to this, Socket.io is has about 1/3 of the performance (~ 2300 vs ~8000 connections) when using WebSockets.

Second, it appears that 0.8.7 is about 20% slower than 0.6.17 on this benchmark. If I remember correctly, Socket.io 0.7 switched to a new protocol, and there are clearly some performance improvements over 0.7.11 in 0.8.7 (+100 connections in this bench); it's just that the overall performance is still worse in this benchmark than in the old 0.6.17 branch.

Working towards higher-performance

As this is just a simple benchmark, I don't really have solutions - only some suggestions.

1) A CI build that includes benchmarks and community contributed test cases

First, I'd love to see a CI build for Socket.io that would include performance benchmarks and community contributed test cases.

However, currently setting up a CI build for Socket.io is difficult because the bundled test suite only works on OSX. It would be a lot easier to contribute if the tests worked on other platforms.

I am hoping that as Engine.io gets going, the test suite will be fixed so that it can be run on other platforms. Otherwise, contributing improvements will be tricky/impossible since there is no way to tell whether the code works.

2) More realistic performance test scenarios

The current test scenario is rather limited in that it mostly tests performance in terms of establishing connections (without terminating them). I'd love to hear more realistic scenario suggestions, particularly from people who have run into memory usage issues.

siobench is only a starting point: it's way better than just looking at htop and wondering whether performance was better in the last version or not. There are still specific questions that should be formulated as replicable tests.

3) A polling transport that works on Node.js

I did write tests for the xhr-polling transport for Socket.io as well. These showed much worse performance, around:

~ 550 connections on Socket.io 0.6.17 (vs ~2300 using WS)
~ 450 connections on Socket.io 0.8.7 (vs ~ 1900 using WS)

+However, the xhr-polling is severely broken in that it stops connecting after 4-5 connections on Node v0.4.12. So I had to force each load generating client to only make four connections and then spawn a new load generating process to work around the problem. I wouldn't vouch for the accuracy of the test with xhr-polling until the xhr-polling transport is fixed on Node when using socket.io-client (it's been broken for the last three releases, though).

4) Comparative benchmarks

Hopefully, this will help with performance testing new releases of Socket.io and other Comet libraries. Since the plan is that Engine.io will allow people to work with a lower level than Socket.io, there might be new performance oriented versions, and it would be useful to see benchmarks for those. Re: the other Node.js pubsub frameworks: I can't benchmark Faye, because it does not provide the right API out of the box, and Juggernaut uses Socket.io internally.

I'm going to use siobench it for internal testing to ensure that the pubsub implementation I am working on (built over Socket.io) will not have performance regressions.

The full graphs are below. Please leave comments and suggestions for improvements - I am hoping that the developer community around Socket.io can help in improving the performance going forward, kind of like what Mozilla did with "arewefastyet.com".

Socket.io 0.6.17 - Websockets - CPU usage and time

Socket.io 0.6.17 - Websockets - resident set size

Comments

Xananax: Hey Mixu. My comment is totally unrelated to your post (which has been useful to me by the way). You don't know me, but I know you a little: I have been following your blog for a while now. But I never had a compelling reason to write something. +So even if it messes up a bit with the rigor of coder's comment that should always be constructive, I wanted to express my thanks as your writings have been an invaluable source of information and inspiration to me. Always interesting, always well written, always well presented. Keep up the good work! +Wish you the best for this new year.

Mikito Takada: Thanks! This comment made my day. 2012 will be a great year!

Karen: You can check with this: https://github.com/sockjs/sockjs-node ?

Luc: Hi,

I've tried to fetch your project on github and when I do "npm install" I've got an error fetching dependencies :

npm ERR! git clone git@github.com:mixu/nodeunit-runner.git Warning: Permanently added 'github.com' (RSA) to the list of known hosts. +npm ERR! git clone git@github.com:mixu/nodeunit-runner.git Permission denied (publickey). +npm ERR! git clone git@github.com:mixu/nodeunit-runner.git fatal: The remote end hung up unexpectedly +npm ERR! Error: git "clone" "git@github.com:mixu/nodeunit-runner.git" "/tmp/npm-1330952664150/1330952664173-0.6050085364840925" failed with 128 +npm ERR! at ChildProcess. (/usr/lib/nodejs/npm/lib/utils/exec.js:49:20) +npm ERR! at ChildProcess.emit (events.js:70:17) +npm ERR! at maybeExit (child_process.js:361:16) +npm ERR! at Socket. (child_process.js:466:7) +npm ERR! at Socket.emit (events.js:67:17) +npm ERR! at Array.0 (net.js:320:10) +npm ERR! at EventEmitter._tickCallback (node.js:192:40)

Do you have any ideas ?

rufinus: just change the github rep url to git://github.com/sockjs/sockjs-node.git in the config file for the npm he used his personal read/write url.

thanks for this article, its exactly what we have seen and couldnt believe it.

My Arch Linux setup

Mixu — Thu, 10 Nov 2011 08:00:00 GMT

This is mostly just a reminder for myself - but I always learn new things when I read how other people set up their system. Leave a comment if you have a tip - that's how I learned about wicd-gtk :). Oh, and install my window manager (tiling, written in C++ and node.js, configurable using Javascript).

First steps

do the basic arch setup first (or VMware, or Virtualbox)

+Update the system (and install/setup sudo)

dhcpcd eth0 #if you did not add "interface=eth0" in rc.conf during setup
+pacman -Syu

Fixes:

http://www.archlinux.org/news/initscripts-update-manual-intervention-required/

rm /etc/profile.d/locale.sh

http://www.archlinux.org/news/filesystem-upgrade-manual-intervention-required/

pacman -S filesystem --force

pacman -S sudo
+vim /etc/sudoers # add yourself to sudoers
+sudo vim /etc/pacman.conf # set SigLevel = Never TrustAll
+sudo shutdown-r now

Install X11:

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils xterm

If virtualized in VirtualBox, make copy-paste work first:

pacman -S virtualbox-archlinux-additions

Details

Create a new user

pacman -S zsh
+useradd -m -g users -G audio,lp,optical,storage,video,games,power,scanner -s /bin/zsh USERNAME
+su USERNAME
+passwd

Add x11

pacman -S xorg-server xorg-xinit xorg-utils xorg-server-utils

Copy ssh keys over from old machine +Useful packages

pacman -S base-devel sudo python2 git libev mlocate mercurial nitrogen
+sakura wicd-gtk pcmanfm gnome-icon-theme htop unzip
+openssl chromium flashplugin bash-completion xterm
+epdfview mysql ruby tilda tmux wget redis xcursor-vanilla-dmz
+xarchiver gzip bzip2 zip unzip unrar p7zip
+meld ttf-ubuntu-font-family mpg123 alsa-utils redis mysql ruby libxslt

base-devel and python2 for compiling node
libev for nwm
mlocate for locate command
nitrogen is better than feh for multiple screens
sakura is a nice terminal
wicd-gtk is a simple wifi network gui
pcmanfm gnome-icon-theme are for pcmanfm, a Nautilus alternative

+Remember to visudo and remove the password requirement from wheel. And add dbus and wicd to /etc/rc.conf just like pacman tells you to. +Configuring X11

Add ~/.Xresources:

Xcursor.theme: vanilla-dmz
+Xcursor.size:  16       !  32, 48 or 64 may also be good values

Configuring git

git config --global color.ui true

Configuring sakura

I want to use ctrl + Page_Up / Page_Down to switch tabs, so edit ~/.config/sakura/sakura.conf:

switch_tab_accelerator=4 # since GDK_CONTROL_MASK is 1 << 2, e.g. 4.
+prev_tab_key=Page_Down
+next_tab_key=Page_Up

Some basic niceties: whatprovides, service and chkconfig

pacman -S pkgtools

pkgtools provides the pkgfile tool. It works like yum whatprovides (e.g. allows you to search for a particular command or dependency in all the pacman packages)
"sudo pkgfile -u" to update the db
"pgkfile zipinfo" to search for zipinfo

+Arch don't have a service and chkconfig, but we can make the new things curve a bit less steep by adding some functions to .bashrc:

function service() {
+  sudo /etc/rc.d/$1 $2
+}
+
+alias chkconfig='cat /etc/rc.conf | grep DAEMONS && echo "cat + grep /etc/rc.conf"'

This makes service an alias for /etc/rc.d/ and prints out the enabled services from /etc/rc.conf. While we're editing .bashrc, might as well add:

PS1="[W]$ " # my preferred bash prompt (e.g. only the current dirname).
+ulimit -s 16400 # higher stack limit
+# ssh-agent
+SSH_ENV="$HOME/.ssh/environment"
+function start_agent {
+     echo "Initialising new SSH agent..."
+     /usr/bin/ssh-agent | sed 's/^echo/#echo/' &gt; "${SSH_ENV}"
+     echo succeeded
+     chmod 600 "${SSH_ENV}"
+     . "${SSH_ENV}" &gt; /dev/null
+     /usr/bin/ssh-add;
+}
+# Source SSH settings, if applicable
+if [ -f "${SSH_ENV}" ]; then
+     . "${SSH_ENV}" &gt; /dev/null
+     #ps ${SSH_AGENT_PID} doesn't work under cywgin
+     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ &gt; /dev/null || {
+         start_agent;
+     }
+else
+     start_agent;
+fi

Installing Node and NPM

You can just do:

pacman -S nodejs

If you're OK with that version, which seems to track the Node releases pretty well.

Arch uses python3 as python. You need to change python to python2 (thanks Rob Searles!)

# node.js fix for arch (use python2) 
+mkdir /tmp/bin
+ln -s /usr/bin/python2 /tmp/bin/python
+export PATH=/tmp/bin:$PATH

You can then do a regular node install:

git clone git://github.com/joyent/node.git

git checkout v0.4.12

./configure

make

sudo make install

Remember to install npm as well:

curl http://npmjs.org/install.sh | sudo sh

Installing my window manager and personal config

git clone git://github.com/mixu/nwm.git
+cd nwm
+node-waf clean || true && node-waf configure build
+sudo npm link # add a global npm symlink to this repository - so nwm-user can find it (man npm link)
+cd ..
+git clone git://github.com/mixu/nwm-user.git
+cd nwm-user
+npm link nwm # now make a symlink to the nwm installation

Add it to ~/.xinitrc (change paths!!):

exec /usr/local/bin/node ~/mnt/nwm-user/nwm-user.js 2&gt;~/nwm.err.log 1&gt;~/nwm.log

And while we're at it, lets add some other stuff:

VBoxClient-all &
+export PATH=/tmp/bin:$PATH # for node-waf, too lazy to work on a better solution
+xset +fp /usr/share/fonts/local
+xset fp rehash

Run "startx" to start X11 with nwm.

Installing my mp3 player

First, we need to configure alsa (included by default):

pacman -S mpg123 alsa-utils

Run:

alsamixer

and turn on Master and PCM channels (by pressing m) as they are muted by default.

sudo alsactl store

Then continue:

git clone git://github.com/mixu/nplay.git

Run nplay with

node nplay.js

TODO: fix directory in source code and change backend from mpg321 to mpg123.

Switching the keyboard language in X11

I sometimes need to write emails in Finnish, so here is how to switch the layout:

<code>setxkbmap -layout fi # revert back code><span class="Apple-style-span" style="font-family: monospace;">setxkbmap -layout usspan><span class="Apple-style-span" style="font-family: monospace;"> span>

Install yaourt

Install dependencies

yaourt libpng12 gtk2-theme-dust

Install Sublime Text 2

Sublime Text needs libpng12, which you have to install from AUR:

wget http://aur.archlinux.org/packages/li/libpng12/libpng12.tar.gz
+tar -xzvf libpng12.tar.gz
+cd libpng12
+makepkg
+pacman -U ./libpng12-1.2.46-2-x86_64.pkg.tar.xz

Then download and run Sublime Text 2.

Also, you might want to ger http://aur.archlinux.org/packages/gt/gtk2-theme-dust/gtk2-theme-dust.tar.gz.

Configuring Sublime Text 2

locate Packages # returns ~/.config/sublime-text-2/Packages
+cd ~/.config/sublime-text-2/Packages
+git clone https://github.com/buymeasoda/soda-theme/ "Theme - Soda"
+cd User
+wget http://blog.mixu.net/files/2010/05/my_themes.zip # Install my themes
+unzip my_themes
+rm my_themes.zip

Base File settings

{ + // FONTS and COLORS + "color_scheme": "Packages/User/Mixu Espresso.tmTheme", + "font_size": 11, + "tab_size": 2, + // WHITESPACE + // Set to true to insert spaces when tab is pressed

Comments

Kenneth: What are the advantages of your window manager? I'm currently using awesome wm on arch. The javascript configuring sounds cool.

Mikito Takada: Mostly: configurability and the ability to customize things like layouts in JS. And you don't need any additional tech beyond Node + npm, so if you are already doing Node dev it's a lightweight option.

jiyinyiyong: As I failed to install Xmonad on Ubuntu 12.04 , I tried NWM, really concise WM. I was supposing to get a WM like Xmonad with Gnome whose fonts and menus look more beautiful(maybe get panels as well..). I've went through the docs on the site, but is there more detail about making NWM look better?

Mikito Takada: nwm works with both Gnome and KDE apps, but it's a window manager, not a desktop environment.

http://en.wikipedia.org/wiki/Desktop_environment vs http://en.wikipedia.org/wiki/Window_manager

E.g.:

Font rendering in X11 is done by libraries (e.g. freetype2 for TTF), not the window manager.
Menu rendering in X11 is done by the window toolkit that the particular app is compiled against e.g. Gnome/KDE, not the window manager.

I'd love to write all that, but I'm just one dude. Writing a desktop environment is a bit too much for a for-fun project ;).

For better looking fonts, I use the Ubuntu font rendering patchset for Arch; for better menus I use a Gnome theme.

jiyinyiyong: (I'm late.. but didn't received e-mail about the reply.. ) +OK, I'll make effort to get familiar about X11 and things like that. So far I just want to try cool stuffs writtern in Node. Really thanks.

jsc: You probably found a way by now, but if not, I would like to mention gtk-theme-switch. It's in the default repos of 12.04 and requires no logout to take effect.

It does what it says on the box: changes your gtk theme to any other one you have installed, which on 12.04 should be at least Ambiance and Radiance.

Nginx, Websockets, SSL and Socket.IO deployment

Mixu — Sat, 13 Aug 2011 07:00:00 GMT

I've spent some time recently figuring out the options for deploying Websockets with SSL and load balancing - and more specifically, Socket.IO - while allowing for dual stacks (e.g. Node.js and another dev platform). Since there seems to be very little concrete guidance on this topic, here are my notes - I'd love to hear from you on your implementation (leave a comment or write about and link back)...

The goal here is to:

Expose Socket.io and your main application from a single port -- avoiding cross-domain communication
Support HTTPS for both connections -- enabling secure messaging
Support the Websockets and Flashsockets transports from Socket.io -- for performance
Perform load balancing for both the backends somewhere -- for performance

+Socket.io's various transports

Socket.io supports multiple different transports:

WebSockets -- which are essentially long lived HTTP 1.1 requests, which after a handshake upgrade to the Websockets protocol
Flash sockets -- which are plain TCP sockets with optional SSL support (but Flash seems to use some older SSL encryption method)
various kinds of polling -- which work over long lived HTTP 1.0 requests

+ +

Starting point: Nginx and Websockets

+ +

Nginx is generally the first recommendation for Node.js deployments. It's a high-performance server and even includes support for proxying requests via the HttpProxyModule.

However, -- and this should be made much obvious to people starting with Socket.io -- the problem is that while Nginx can talk HTTP/1.1 to the client (browser), it talks HTTP/1.0 to the server. Nginx's default HttpProxyModule does not support HTTP/1.1, which is needed for Websockets.

Websockets 76 requires support for HTTP/1.1 as the handshake mechanism is not compatible with HTTP/1.0. What this means is that if Nginx is used to reverse proxy a Websockets server (like Socket.io), then the WS connections will fail. So no Websockets for you if you're behind Nginx.

There is a workaround, but I don't see the benefit: use a TCP proxy (there is a custom module for this by Weibin Yao, see here ). However, you cannot run another service on the same port (e.g. your main app and Socket.io on port 80) as the TCP proxy does not support routing based on the URL (e.g. /socket.io/ to Socket.io and the rest to the main app), only simple load balancing.

So the benefit gained from doing this is quite marginal: sure, you can use Nginx for load balancing, but you will still be working with alternative ports for your main app and Socket.io.

Alternatives to Nginx

+Since you can't use Nginx and support Websockets, you'll need to deal with two separate problems:

How to terminate SSL connections and
How to route HTTP traffic to the right backend based on the URL / load balance

+If you want to run two services on the same port, then you will have to terminate SSL connections before doing anything else. There are several alternatives for SSL termination:

Stunnel. Supports multiple SSL certificates per process, does simple SSL termination to another port.
Stud. Only supports one SSL certificate per invocation, does simple SSL termination to another port.
Pound. An SSL-termination-capable reverse proxy and load balancer.
Node's https. Can be made to do anything, but you'll have to write it yourself.

+If you choose Stunnel or Stud, then you need a load balancer as well if you plan on having more than one Node instance in the backend.

HAProxy is not generally compatible with Websockets, but Socket.IO contains code which works around this issue and allows you to use HAProxy. This means that the alternatives are:

Stunnel for SSL termination + HAProxy for routing/load balancing
Stud for SSL termination + HAProxy for routing/load balancing
Pound (SSL and routing/load balancing)

+I haven't looked into Pound more - mainly as I could not find info on it's TCP reverse proxying capabilities (see the section on Flash sockets below), but it seems to work for these guys.

Setting up Stunnel

+The Stunnel part is quite simple:

cert = /path/to/certfile.pem
+; Service-level configuration
+[https]
+accept  = 443
+connect = 8443

If you only have one Node instance, you can skip setting up HAProxy, since you don't need load balancing.

Setting up HAProxy

+Would you like Flash Sockets with that?

Note that we need TCP mode in order to support Flash sockets, which do not speak HTTP.

Flash sockets are just plain and simple TCP sockets, which will start by sending the following payload: ''. They expect to receive a Flash cross domain policy as a response.

Since Flash sockets don't use HTTP, we need a load balancer which is capable of detecting the protocol of the request, and of forwarding non-HTTP requests to Socket.io.

HAProxy can do that, as it has two different modes of operation:

HTTP mode - which allows you to specify the backend based on the URI
TCP mode - which can be used to load balance non-HTTP transports.

+Main frontend

We accept connections on two ports: 80 (HTTP) and 8443 (Stunnel-terminated HTTPS connections).

By default, everything goes to the backend app at port 3000. Some HTTP paths are selectively routed to socket.io

TCP mode is needed so that Flash socket connections can be passed through, and all non HTTP connections are sent to the TCP mode socket.io backend.

# Main frontend
+frontend app
+  bind 0.0.0.0:80
+  bind 0.0.0.0:8443
+  # Mode is TCP
+  mode tcp
+  # allow for many connections, with long timeout
+  maxconn 200000
+  timeout client 86400000
+
+  # default to webapp backend
+  default_backend webapp
+
+  # two URLs need to go to the node pubsub backend
+  acl is_socket_io path_beg /node
+  acl is_socket_io path_beg /socket.io
+     use_backend socket_io if is_socket_io
+
+   tcp-request inspect-delay 500ms
+   tcp-request content accept if HTTP
+   use_backend sio_tcp if !HTTP

Port 843: Flash policy

Flash policy should be made available on 843.

# Flash policy frontend
+frontend flashpolicy 0.0.0.0:843
+   mode tcp
+   default_backend sio_tcp

Default backend

This is just for your main application.

backend webapp
+   mode http
+   option httplog
+   option httpclose
+   server nginx1s localhost:3000 check

Socket.io backend

Here, we have a bunch of settings in order to allow Websockets connections through HAProxy.

backend socket_io
+  mode http
+  option                  httplog
+  # long timeout
+  timeout server 86400000
+  # check frequently to allow restarting
+  # the node backend
+  timeout check 1s
+  # add X-Forwarded-For
+   option forwardfor
+  # Do not use httpclose (= client and server
+  # connections get closed), since it will close
+  # Websockets connections
+  no   option httpclose
+  # Use "option http-server-close" to preserve
+  # client persistent connections while handling
+  # every incoming request individually, dispatching
+  # them one after another to servers, in HTTP close mode
+  option http-server-close
+  option forceclose
+  # just one node server at :8000
+  server node1 localhost:8000 maxconn 2000 check

Socket.io backend in TCP mode

Comments

Tom: How are you handling HTTP requests then?

Mikito Takada: As configured above, HAProxy listens at two ports:

80 (e.g. for direct HTTP requests) +8443 (e.g. for SSL-terminted HTTP requests)

And the default HAProxy backend is the main app.

HTTP comes in at port 80 to HAproxy. +HTTPS comes in at port 443 -> Stunnel terminates SSL -> 8443

Andrew: Hi, we've been integrating support for http/1.1 to backends for nginx lately. Would you like to test your setup with http/1.1 development code? Would be much appreciated.

mikong: Hi Andrew,

We're working on NodeJS apps that needs Websockets, and thanks to your comment, we'll be taking a look at the development version of Nginx. We'll send you feedback for any problems.

Patrick: Hi there,

I'm trying to setup something similar. I have a rails app on port 3000, and a nodejs app on port 3001. The nodejs app doesnt only do sockets though... it also has a little http api using Express.

I wanted to have a different subdomain for each app, and use haproxy to serve the right thing but I can't seem to see how to do that... seems like when I use tcp mode I can't really do anything regarding the domain that was requested.

Mikito Takada: Yeah, I found the tcp mode somewhat limited in Haproxy. You might want have a look at node-http-proxy...

Patrick: I tried that first actually, for some reason I could not get node-http-proxy to work for Chrome PC and Internet Explorer. Or at least, socket.io seems to fall back to polling transports for some browsers when behind node-http-proxy. But maybe my configuration was wrong...

Joe Lindsay: We need to solve this problem, how can I get the right bytes to test http/1.1 backends to nginx?

theTestTube: Maybe the following directive can let you deal with "[...] The nodejs app doesnt only do sockets"

acl websocket hdr(Upgrade) -i WebSocket

I found this at http://stackoverflow.com/questions/2419346/can-nginx-be-used-as-a-reverse-proxy-for-a-backend-websocket-server

Luke: This configuration does not work for me. According to HAProxy docs you cannot use layer 7 filtering with TCP mode. On my setup this caused to use random backend.

I managed to fix this configuration in HTTP mode. The main was flash-policy-file serving - it is not served if socket.io client manage to get the policy from the same server on port 843.

If you provide policy file on that port then flash will not request policy file and use standard HTTP headers for communication. This makes HAProxy happy.

My config is here: +http://pastebin.com/g8KQSWTW

Many thanks for your post - it was the best starter to make websockets, xhr-pooling and flashsockets to work with HAProxy.

Pau ramon: I use substacks bouncy https://github.com/substack/bouncy

Mike Lewis: With newer versions of nginx you can use 1.1 for reverse proxying.

This seems to work for reverse proxying websocket connections:

location / {
+    chunked_transfer_encoding off;
+    proxy_http_version 1.1;
+    proxy_pass        http://localhost:9001;
+    proxy_buffering   off;
+    proxy_set_header  X-Real-IP  $remote_addr;
+    proxy_set_header  Host $host:9001;
+    proxy_set_header  Connection "Upgrade";
+    proxy_set_header  Upgrade websocket;
+}

Jimb Esser: I did a bunch of experimenting and some digging, and it appears there is not currently any way to get nginx to work with websockets using a method like that. WebSockets needs more than just forwarding a request, and the nginx developers say more work would need to be done to get it proxying websockets. You can, however, use nginx with the tcp proxying module to blindly load-balance, and then put stunnel on each of your application servers, though that removes most of the benefit of nginx.

Ximonn: Why not do it simple and use two servers. Then configure dns to point socket.mydomain.com to the websocket server and mydomain.com to the nginx server. +You use different IP's but the same domain name, sofar I havent seen any issue with that.

Edgar: Trying to load balance several node.js servers but can't make it work. +When using more than one server, the browser seems to keep reconnecting. +Everything is great when using just one node.

The haproxy conf I'm using is the following. +Am I missing anything?

backend socket_io + balance roundrobin

mode http
+option                  httplog
+
+# long timeout
+timeout queue 5000
+timeout server 86400000
+timeout connect 86400000
+
+# check frequently to allow restarting
+# the node backend
+timeout check 1s
+
+# add X-Forwarded-For
+option forwardfor
+
+# Do not use httpclose (= client and server
+# connections get closed), since it will close
+# Websockets connections
+no   option httpclose
+
+# Use "option http-server-close" to preserve
+# client persistent connections while handling
+# every incoming request individually, dispatching
+# them one after another to servers, in HTTP close mode
+option http-server-close
+option forceclose
+
+
+# balance
+server node1 127.0.0.1:8001 weight 1 maxconn 2000 check
+server node2 127.0.0.1:8002 weight 1 maxconn 2000 check

Mikito Takada: Hi Edgar,

You can't round robin Socket.io - it's a protocol spec limitation.

I've written a follow up to this post on my book, just skip to the section "Sample deployments: scaling" at http://book.mixu.net/ch13.html

Edgar: Hi,

You saved my day. +Thanks.

James: I believe that would run amok of SOP restrictions

Ram: How about node-http-proxy - https://github.com/nodejitsu/node-http-proxy

coryvirok: Great post! I setup the same deployment for https://ratchet.io/ earlier this year. We are, however running into some trouble with stunnel not setting the x-forwarded-for header. Looks like stunnel supports something like this via the transparent=source config option however our server's kernel doesn't have the necessary flags enabled.

Are you able to get the incoming client IP from your node.js or ruby servers?

Axel: Another way to deploy Socket.IO is to use Kaazing's HTML5 Gateway. Kaazing will proxy the Socket.IO traffic and it takes care of the scaling, emulation, clustering, security and even provides an SSO integration. The Gateway is basically like a network router, except it's protocol aware. Kaazing does all the emulation if WS is not available in a particular browser so you don't have to worry about getting polling and Flash-Socket to work. In fact, the polling fall back from Socket.IO is really slow, and also seriously increases the battery consumption on mobile apps. And Flash Socket is notoriously insecure. Kaazing will emulate WS all the way back to an old IE6 browser as well, and doesn't have the firewall traversal issues that Socket.IO's alternate transport layers have.

All you have to do is to put Node.JS behind the Gateway and use only the WS transport. You just use the Kaazing client library below the Socket.IO client library and you make sure the client also uses the WS transport only. The Gateway will readily take care of all production needs from there on. Their Cloud solution is relatively cheap, you can just rent an EC2 instance, the cost is a tiny fraction on top of what EC2 charges, and it adds up to much less than some of the basic publish/subscribe service providers out there. And there is no lock in because the entire Kaazing stack is based on the WS standard, so you just program against the WS APIs.

Now a disclaimer: I work for Kaazing, but I am not posting this on behalf of Kaazing. I happen to be a major fan and enthousiast of Node.JS, and I would like to see more Node.JS adoption in large enterprises. Kaazing unfortunately doesn't currently market this solution at all. Using Kaazing to deploy Node.JS just makes Node.JS so much more attractive to enterprise shops because it overcomes any and all objections anyone would have against using Node.JS in an enterprise.

Asbestos: Pound doesn't work. +Nginx should have websocket compatibility by Jan 2012 - http://trac.nginx.org/nginx/roadmap

suff: HAProxy from version 1.5dev12 (currently 1.5dev16) supports SSL.

Collaborative git reference

Mixu — Sat, 06 Aug 2011 07:00:00 GMT

Here is a basic reference for collaborative git commands:

Checkout a remote branch

+ +

# list branches first
+git branch -a
+# * master
+# remotes/origin/branch_name
+git checkout -b local_name remotes/origin/branch_name

Merging

+ +

# switch to master
+git checkout master
+# merge with experimental branch
+git merge experimental

Create a tag

+ +

# list tags
+git tag
+# add a tag
+git tag tagname
+# push to remote
+git push origin master --tags

dwm tips on Fedora

Mixu — Sat, 11 Jun 2011 07:00:00 GMT

I've been testing out Fedora 15's Gnome 3 and Ubuntu's Unity, and didn't like either of them. They both take up too much precious screen space just to show a fancy UI, and requiring hardware acceleration is a pain for low end netbooks and virtual machines.

So I decided to move to an alternative window manager. DWM (dynamic window manager, http://dwm.suckless.org/) is an extremely lightweight tiling window manager written in C which saves screen space and works pretty well as long as you don't need to connect to wireless networks.

I've been pretty happy with it. The main drawback is that connecting to wireless networks is a pain in the ass as there are no proper GUI tools to do this. Check out these tips to get started with dwm

0. Installing dwm on Fedora, keyboard shortcuts

+To install DWM, run yum install dwm. You can then choose to use dwm or Gnome or Kde in the login screen. + +The default keyboard shortcuts are listed at man dwm or at http://man.suckless.org/dwm/1/dwm. +

1. Customizing dwm

+Customizing dwm can be done by making changes to config.h and recompiling the window manager. + +Fedora has a really nice package called dwm-user, which automates this process! Here is the package description: + +dwm-start is a helper script for running and reconfiguring dwm if neccessarry. It's the preferred way of starting dwm in Fedora. +Running dwm-start starts Fedora build by default. If you wish to customize your configuration, put the dwm config header file to $HOME/.dwm/config.h and adjust it according to your needs. Every time the user configuration file has changed, dwm-start will rebuild the user dwm binary prior to its execution. + +All you need to do is: + +

bash
+sudo yum install dwm-user

+ + +

bash
+mkdir ~/.dwm

+ + +

bash
+cp /usr/src/dwm-user-5.8.2-6.fc14/config.def.h ~/.dwm/config.h

+ +E.g. install via yum, then make a ~/.dwm folder, then copy the config.h file and edit it. When you restart, you can choose dwm-user as your window manager which uses you custom version of dwm. For example, I remapped Meta (Cmd/Windows key) + h and meta + l to meta + pg up / pg down and meta + shift + q to meta + shift + end since I'm currently running Fedora on an OSX host. + +You will probably make changes to the keyboard shortcuts. To find the keymap: + +

bash
+sudo updatedb

+ + +

bash
+locate keysymdef.h

+ +Keysymdef.h lists the names of the keys in X11. +

2. Tip: Guake is just as awesome on dwm

+By default, dwm launches xterm. I prefer to use guake, since that allows me to get the tabbed terminal window on any workspace when I need them. Just launch guake& to run it in the background. + +UPDATE: I moved to F15 (deciding simply to ignore Gnome 3) and noticed that guake has problems starting. To fix those: + +

bash
+sudo yum install xfce4-notifyd

+ +Basically you need a notify daemon to allow Guake to print that pretty message "Guake is running", and xfce4-notifyd provides an alternative notifications daemon. +

3. Launch netbeans and other Java programs with font smoothing and GTK look and feel

+You need to specify a couple of extra switches to get the GTK look and feel in Java programs, for example: + +/home/username/netbeans-7.0/bin/netbeans -J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=on --laf com.sun.java.swing.plaf.gtk.GTKLookAndFeel +

4. Launch nautilus without the desktop

+ +

nautilus --no-desktop

5. Use dwm with a dual screen setup

+If dwm starts with mirroring output to your secondary screen, then you need to run xrandr to get the names your screens. E.g. VBOX0 and VBOX1. + +Then configure the screen layout: + +

bash
+xrandr --output VBOX1 --auto --right-of VBOX0

+ +dwm will now let you have your own workspaces for each screen. +

6. Change your desktop background

+Use feh to change your desktop background: + +

bash
+feh --bg-tile /path/to/background/image

+ +

7. xterm config

+For a usable xterm, create the following ~/.Xresources and run + +

bash
+xrdb -merge .Xresources

+ + +

bash
+xterm*faceName:           monospace:pixelsize=14
+xterm*saveLines:          9999
+xterm*scrollBar:          false
+xterm*background:  #000000
+xterm*foreground:  #dfdfdf
+xterm*color0:      #000000
+xterm*color1:      #9e1828
+xterm*color2:      #aece92
+xterm*color3:      #968a38
+xterm*color4:      #414171
+xterm*color5:      #963c59
+xterm*color6:      #418179
+xterm*color7:      #bebebe
+xterm*color8:      #666666
+xterm*color9:      #cf6171
+xterm*color10:     #c5f779
+xterm*color11:     #fff796
+xterm*color12:     #4186be
+xterm*color13:     #cf9ebe
+xterm*color14:     #71bebe
+xterm*color15:     #ffffff

+ +

8. Add a clock using xsetroot

+You can do something like this in a bash script to show the time in dwm on the top right corner. + +

bash
+
+

while true; do + xsetroot -name "$( date +"%F %R" )" + sleep 1m # Update time every minute +done + +

9. Connect to wifi

+This is rather painful. The instructions here were collected from the mailing list, and I did get them to work, but I'm too lazy to write a full tutorial on this right now. + +Basically, you need to scan, then do different things depending on whether the wifi uses WEP or WPA for authentication. + +Start by running: + +bash +iwlist scan + +

9.1 WEP wifi

+ +

&gt; &gt; #wep connect to a wep wifi
+&gt; &gt; #! /bin/sh
+&gt; &gt;
+&gt; &gt; key="grep $1 /home/pmarin/wep | cut -d' ' -f2"
+&gt; &gt; sudo ifconfig wlan0 up
+&gt; &gt; sudo iwconfig wlan0 essid $1
+&gt; &gt; sudo iwconfig wlan0 key s:$key
+&gt; &gt; sudo dhclient wlan0
+&gt; &gt; #end

&gt; &gt; The wep is a plain file with to columms
+&gt; &gt;
+&gt; &gt; essid  key

9.2 WPA wifi

+ +

&gt; &gt;
+&gt; &gt; #wpa connect to a wpa wifi
+&gt; &gt; #! /bin/sh
+&gt; &gt;
+&gt; &gt; sudo ifconfig wlan0 up
+&gt; &gt; sudo iwconfig wlan0 essid $1
+&gt; &gt; sudo wpa_supplicant -iwlan0 -c/home/pmarin/wpa -B
+&gt; &gt; sudo dhclient wlan0
+&gt; &gt; #end
+&gt; &gt;
+&gt; &gt; the wpa file is similar than /etc/wpa_supplicant.conf

To create the wpa file:

wpa_passphrase your_ssid_of_network your_network_password

Create the file:

ctrl_interface=/var/run/wpa_supplicant
+#ap_scan=2
+
+network={
+       ssid="your_ssid"
+       scan_ssid=1
+       proto=WPA RSN
+       key_mgmt=WPA-PSK
+       pairwise=CCMP TKIP
+       group=CCMP TKIP
+       psk=your_psk_from_wpa_passphrase
+}
+
+sudo wpa_supplicant -Bw -Dwext -i eth0 -c/etc/wpa_supplicant.conf

9.3 Wifi troubleshooting:

+1) CHECK THAT YOU DON'T have the NetworkManager service or wpa_supplicant running already!!!

You can run wpa_supplicant with -dd flag for a detailed debug output.1) If you don't manage to connect to the AccessPoint, try to uncomment line 2 in /etc/wpa_supplicant.conf.

2) If that doesn't help, try change its value to 0 or 1.

3) If you get troubles while authenticating, try removing "RSN" and/or"CCMP" strings from /etc/wpa_supplicant.conf.

Sources for Wifi stuff:

http://ubuntuforums.org/showthread.php?t=263136

http://www.mail-archive.com/dwm@suckless.org/msg06800.html

Comments

vimarsh: One tip for the wifi set-up: i found it easier to set up the initial config using nm-connection-editor and then start nm-applet in the dwm launcher script. It will autoconnect to your preferred wifi configuration. (Of course you need NetworkManager daemon running)

Sample launcher:

""" +#

start new dwm session

nm-applet --sm-disable& +exec /usr/local/bin/dwm +"""

Mikito Takada: Thanks for the tip!

Matt Freeman: In my .xsession I have 'nm-applet &' (or perhaps you can spawn it from the dwm config), if this is setup correctly then it will auto connect to wifi network previous configured (i.e. the one you setup whilst in gnome). Alternatively if you have a tray program such as trayer or tint2 etc.. you can work with the network manager just as you would in Gnome.

Hardcore Arch users tell me that wicd is better alternative without zero gnome attachments though. - not verified

Mikito Takada: Thank you, I gave wicd a try and it seems like a really nice solution for going Gnome-free...

sudo yum install wicd wicd-gtk wicd-curses +sudo service NetworkManager stop # Bad password errors if both are running... +sudo service wicd start +wicd-client -n

Anton: Hi! Big thanks for tips! +Always wanted to try tiling managers and with releases of unity and gnome 3, the time to do it have come. now i am messing around lxde based fedora + dwm, and your article is really good point to start.

Hello San Francisco!

Mixu — Mon, 06 Jun 2011 07:00:00 GMT

If you've been wondering why I haven't been writing much on the blog recently, here is why: I just got my visa to the US and moved to San Francisco for an internship! I'll be here for the next 12 months on that visa. That process + wrapping up my life in Finland took a lot of my free time, the rest went to reviewing a coming-soon Kohana 3 book from Apress for which I'm a technical reviewer, and my writing project related to Node.js which I am hoping to move forward soon. I also gave a talk at Frontend.fi on Node (my first tech talk), and released a small library (node-winamp) to control Winamp over LAN from the console using Node.

My new years resolution was to ship more, but I never expected to ship my own ass over to San Francisco! It'll be awesome.

I flew in on June 3rd, and have been here for two days. So far, everything has been pretty awesome, though I still need to find an apartment rental and get the final paperwork done on this side!

Since I'll be able to concentrate more on code instead of living the triple life of a doctoral student, individual entrepreneur and open source programmer, I hope to start building a solid streak of open source/fun coding while I'm here. I read something on HN recently which I think will be good to keep in mind:

"I have a rule: I refuse to go to sleep if I haven’t read and written some code on a given day (this doesn’t include the code I write for work, of course). So far, this rule has had a positive impact on my ability to code."

I'll try to apply that rule, perhaps using 42goals.com or the Calendar About Nothing...

Quick tip: Fix Flash audio stutter on Fedora 14 (64bit)

Mixu — Thu, 24 Feb 2011 08:00:00 GMT

On my FC14 machine, I had a problem with Flash (64bit) audio playback: the sound on sites other than Youtube would stutter terribly. It appears that this a systematic problem; but luckily there is a fix!

Check out Ahmed Abdo's post on Flash audio stutter for the details. Works perfectly for me!

Details: https://bugzilla.redhat.com/show_bug.cgi?id=638477

The bug is triggered by a change in glibc. Who proposed the fix? Linus Torvalds. So I guess the following isn't quite true?

I love the pragmatism from his part:

+So in the kernel we have a pretty strict "no regressions" rule, and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to. + +So I'm disappointed glibc just closes this as NOTABUG. There's no real reason to do the copy backwards that I can see, so doing it that way is just stupid. + +But whatever. You can do a LD_PRELOAD trick to get a sane memcpy(), and it does indeed fix the sound for me. +[...] +The fact that the glibc people don't do that, and that this hasn't been elevated despite clearly being a big usability problem (normal users SHOULD NOT HAVE TO google bugzillas and play with LD_PRELOAD to have a working system), is just sad. +

+ +

Although overall, as an end user it the conversation around this bug and its persistence makes me sad. I know it's selfish not to care about the technical superiority of a solution or about who is to blame here - but I'd just like to have my smooth Flash playback...

Cascading file loading in Node.js

Mixu — Fri, 18 Feb 2011 08:00:00 GMT

One of my favorite features of Kohana 3 is it's cascading filesystem - so I decided to implement it for Node.js. A cascading filesystem is an elegant solution to a common problem: how to provide a mechanism for loading modules and reusing code?

The following image from Kohana 3's docs shows an example:

Benefits

+The key benefits are: +

Consistency. All your application files, including views, controllers, models and other data such as translation messages are loaded using one, easy-to-understand mechanism.
Easy reuse. Without a cascading file system, you'll have to copy and move files around if you want to use someone else's libraries or modules. With a cascading file system, you just place the module in your application, and enable cascading for that directory.
Transparent extensibility. What if you want to override one part of a module (say, a view) but don't want to modify your copy of the module (e.g. so that you can update without manually merging changes). A cascading filesystem allows you to selectively replace files in 3rd party code simply by providing your own version of the file.

The code

https://github.com/mixu/hmvc-cfs/
It's only about 90 lines of code, see this file

Load order and file name resolution

+The load order for my implementation is: +

Application path - files under ./application/ are always checked first.
Module paths - set modules(['./modules/my-module']) to enable module loading. Files from modules are loaded from in the order they are added.
System path - files under ./system/ are loaded if no alternative exists.

+Assumptions about file and class names + +Files are assumed to be lowercase. Underscores in class names are replaced by slashes (so Controller_User becomes ./application/classes/ controller/user.js). + +Performance impact + +Requests are cached, so that additional calls to find_file() do not cause additional stat() calls in the filesystem. This is insignificant anyway, since Node.js servers are persistent so the cascading search is only done once per server instance for each file (not once per request). + +Loading 3rd party code + +The loaded files do not need to be "compatible" in any way other than layout in the file system. For example, while Cfs.factory('some_other_lib') loads the file from ./application/ classes/some/other/lib.js, that file does not actually need to contain a class named some_other_lib; just that it returns something via module.exports. +

Methods

+The methods are: +

Cfs.modules(['./modules/path-to-module']) - set the modules directories to search.
Cfs.find_file(dir, file, ext) - Search each path under dir (e.g. 'classes', 'views') for file (filename) with the extension (ext, default is ".js").
Cfs.factory(class_name) - Return a new instance of the given class after loading the corresponding file from the cascading file system. Note that classes should be in the classes subdirectory.
Cfs.load(class_name) - Return whatever require(file-which-contains-the-class) returns. Useful for extending classes, see below for an example

Example usage:

+ +

var Cfs = require('./cfs.js');
+
+// test class loading:
+// e.g. check ./application/classes/test.js
+// ./modules/modulename/classes/test.js
+// ./system/classes/test.js
+var t = Cfs.factory('test');
+t.run();
+
+// test view loading
+// e.g ./application/views/user/index.html
+// ./modules/modulename/views/user/index.html
+// ./system/views/user/index.html
+fs.readFile(Cfs.find_file('views', 'user/index', '.html'), function (err, data) {
+  if (err) throw err;
+  sys.puts(data);
+});

To set modules:

// set only once, before calling any other functions!
+Cfs.modules([
+         "./modules/testmodule/",
+         "./modules/testmodule2/",
+         ]);

Extending classes:

+ + +

// test extending class (see code in /application/classes/controller/extend.js
+// to see how extension is achieved)
+// e.g. ./application/classes/controller/extend.js
+// ./modules/modulename/classes/controller/extend.js
+// ./system/classes/controller/extend.js
+var t3 = Cfs.factory('Controller_Extend');
+t3.run();
+t3.run_parent();

Note that if you put cfs.js in ~/node_modules/cfs.js, you don't need to specify the path to it... see Modules in node.js docs.

// in extend.js:
+var ControllerExtend = function () {
+}
+// extend the class
+var util = require('util'), Cfs = require('../../../../cfs.js');
+util.inherits(Controller_Extend, Cfs.load('Controller_Base'));
+
+Controller_Extend.prototype.run = function() {
+   console.log("Controller_Extend from testmodule2.");
+};
+Controller_Extend.prototype.run_parent = function() {
+   // run the parent function
+   Controller_Extend.super.prototype.run();
+};
+
+module.exports = Controller_Extend;

Comments

shadowhand: Calling this HMVC is highly misleading. The cascading filesystem has absolutely nothing to do with hierarchical model-view-controller. I strongly recommend that you rename your "HMVC" class to something reasonable, such as "CFS".

Mikito Takada: @shadowhand I agree and apologize if I have misled people. I had greater ambitions for the module at the time, hence the "HMVC style" reference, but I ended up doing other things. I've removed any references to HMVC here (well, except the diagram showing the cascading file system in Kohana but that's CFS rather than HMVC); I'll do github as well next time I actually work on that repo...

Edit: for clarification if someone ends up here: the title of this post used to be "HMVC -style cascading..."