Back

An ode to unsung heroes

Open water

Introduction

I always write about large projects, shiny things I feel particularly proud of. But once in a while, it's nice to recognize how much of that work relies on load-bearing scripts, written quickly to get something else done, and improved over time. I've reached the point where I sometimes take these little scripts for granted on other systems, and even go so far as to ensure they're present on my work machine and my VPS. So, without further ado, let's take a little tour of these useful scripts. And if you have any recommendations on how to improve these further, please let me know.

My local secrets store

The most useful script I have is called key and solves a simple need: how do I keep track of all my API keys and various non-critical secrets? The answer is essentially a wrapper around cryptographic stores on either Linux (secret-tool or pass if available) or Darwin (using a dedicated keychain via security).

The threat model here is to keep a compromised application from snooping on .env files or any other plain-text storage of my credentials and API keys. Access is not restricted beyond the defaults of the underlying tools: any process can just call key. For most use cases, this is all I need, and infinitely better than having sensitive data written anywhere on disk unencrypted.

Usage
$ key set test
# Value for test: <stdin = "value">
# Saved 'test' in Keychain.

$ key list
# test

$ key get test
# value
Source
#!/usr/bin/env bash

set -euo pipefail

usage() {
    echo "Usage: key {get|set|del} <name>" >&2
    echo "       <value> | key set <name>" >&2
    echo "       key list                " >&2
    exit 2
}

[[ $# -ge 1 ]] || usage
command=$1

case "$command" in
    list)
        [[ $# -eq 1 ]] || usage
        name=""
        ;;
    get|set|del)
        [[ $# -eq 2 ]] || usage
        name=$2
        ;;
    *)
        usage
        ;;
esac

read_value() {
    if [[ -t 0 ]]; then
        IFS= read -r -s -p "Value for $name: " value
        echo
    else
        # Read everything so multi-line values (PEM, JSON) survive a pipe.
        value=$(cat)
    fi
    # strip one trailing carriage return left by clipboard tools
    value=${value%$'\r'}
    [[ -n "$value" ]] || { echo "No value provided." >&2; exit 1; }
}

list_keys() {
    security dump-keychain "$1" | awk -v user="$USER" '
  /class: "genp"/ {
    if (in_item && account == user && service != "")
      print service
    in_item = 1
    account = service = ""
    next
  }

  in_item && /"acct"<blob>="/ {
    line = $0
    sub(/^.*"acct"<blob>="/, "", line)
    sub(/".*$/, "", line)
    account = line
  }

  in_item && /"svce"<blob>="/ {
    line = $0
    sub(/^.*"svce"<blob>="/, "", line)
    sub(/".*$/, "", line)
    service = line
  }

  END {
    if (in_item && account == user && service != "")
      print service
  }
'
}

read_secret() {
    local info hex
    if ! info=$(security find-generic-password -a "$USER" -s "$1" -g "$2" 2>&1 >/dev/null); then
        printf '%s\n' "$info" >&2
        return 1
    fi
    case "$info" in
        "password: 0x"*)
            # `security` prints non-printable values as hex; decode them back.
            hex=${info#password: 0x}
            hex=${hex%% *}
            printf '%b' "$(sed 's/../\\x&/g' <<<"$hex")"
            ;;
        *)
            security find-generic-password -a "$USER" -s "$1" -w "$2"
            ;;
    esac
}

if [[ "$OSTYPE" == darwin* ]]; then
    keychain="${KEY_KEYCHAIN:-$HOME/Library/Keychains/keys.keychain-db}"
    login_keychain="$HOME/Library/Keychains/login.keychain-db"

    # A dedicated keychain keeps `key list` fast; its password is stored in the login keychain.
    if [[ ! -f $keychain ]]; then
        keychain_password=$(openssl rand -hex 32)
        security create-keychain -p "$keychain_password" "$keychain"
        security add-generic-password -U -a "$USER" -s key-keychain-password \
            -w "$keychain_password" "$login_keychain"
    fi

    if ! keychain_password=$(security find-generic-password -a "$USER" \
        -s key-keychain-password -w "$login_keychain" 2>/dev/null); then
        echo "No password stored for $keychain; remove the keychain to recreate it." >&2
        exit 1
    fi
    security unlock-keychain -p "$keychain_password" "$keychain"

    case "$command" in
        get)
            read_secret "$name" "$keychain"
            ;;
        set)
            read_value
            security add-generic-password -U -a "$USER" -s "$name" \
                -w "$value" "$keychain"
            echo "Saved '$name' in Keychain."
            ;;
        del)
            if ! security find-generic-password -a "$USER" -s "$name" \
                "$keychain" >/dev/null 2>&1; then
                echo "No key named '$name'." >&2
                exit 1
            fi
            security delete-generic-password -a "$USER" -s "$name" "$keychain" \
                >/dev/null 2>&1 || {
                echo "Failed to delete '$name'." >&2
                exit 1
            }
            echo "Deleted '$name' from Keychain."
            ;;
        list)
            list_keys "$keychain"
            ;;
        *)
            usage
            ;;
    esac
elif [[ "$OSTYPE" == linux* ]]; then
    if command -v secret-tool >/dev/null 2>&1; then
        case "$command" in
            get) secret-tool lookup service key name "$name" ;;
            set)
                read_value
                printf '%s' "$value" | secret-tool store --label="key $name" service key name "$name"
                echo "Saved '$name' in keyring."
                ;;
            del) secret-tool clear service key name "$name" && echo "Deleted '$name' from keyring." ;;
            list) echo "Function not supported on Linux yet" ;;
            *) usage ;;
        esac
    elif command -v pass >/dev/null 2>&1; then
        case "$command" in
            get) pass show "$name" ;;
            set)
                read_value
                printf '%s\n' "$value" | pass insert -m "$name"
                echo "Saved '$name' in pass."
                ;;
            del)
                store="${PASSWORD_STORE_DIR:-$HOME/.password-store}"
                if [[ ! -f $store/$name.gpg ]]; then
                    echo "No key named '$name'." >&2
                    exit 1
                fi
                pass rm -f "$name" >/dev/null
                echo "Deleted '$name' from pass."
                ;;
            list)
                store="${PASSWORD_STORE_DIR:-$HOME/.password-store}"
                if [[ -d $store ]]; then
                    (cd "$store" && find . -type f -name '*.gpg' |
                        sed -e 's|^\./||' -e 's|\.gpg$||' | sort)
                fi
                ;;
            *) usage ;;
        esac
    else
        echo "Linux requires 'secret-tool' or 'pass'." >&2
        exit 1
    fi
else
    echo "Unsupported operating system: $OSTYPE" >&2
    exit 1
fi

Homemade fish bookmarks

Even though I use zoxide instead of cd, I still like to have bookmarks for folders I keep switching between. This is the kind of script you write yourself, because existing tools never quite fit. So here is mine, as a fish function, using universal variables as the backend.

Usage
cd /some/path
bf save "label"
cd ..
bf go label
Source
#!/usr/bin/env fish
#
# Bookmark manager using universal variables in fish

function bf -d "Bookmark manager for directories"

    set -l PREFIX bkm_

    # The subcommand is the first argument (e.g., 'save', 'go')
    set -l subcommand $argv[1]

    # The name of the bookmark is the second argument
    set -l bookmark_name $argv[2]

    # The full universal variable name with our prefix
    set -l var_name "$PREFIX$bookmark_name"

    switch $subcommand
        case save
            if not string match -qr '^[A-Za-z0-9._-]+$' -- "$bookmark_name"
                echo "Invalid bookmark name: $bookmark_name" >&2
                return 1
            end
            set -U "$var_name" (pwd)
            echo "Saved bookmark '$bookmark_name'"

        case go
            if test -z "$bookmark_name"
                echo "Usage: bf go <bookmark_name>"
                # If no name provided, list available bookmarks
                bf list
                return 1
            end

            if set -q "$var_name"
                cd "$$var_name"
            else
                echo "Bookmark '$bookmark_name' not found."
                return 1
            end

        case list
            set -l bookmarks (set -U -n | string match -er "^$PREFIX" | string replace $PREFIX '')
            if test (count $bookmarks) -eq 0
                echo "No bookmarks set."
            else
                echo "Available bookmarks:"
                for name in $bookmarks
                    set -l var "$PREFIX$name"
                    echo "- $name: $$var"
                end
            end
        case del
            if test -z "$bookmark_name"
                echo "Usage: bf del <bookmark_name >"
                return 1
            end

            if set -q "$var_name"
                set -e "$var_name"
                echo "Deleted bookmark '$bookmark_name'."
            else
                echo "Bookmark '$bookmark_name' not found."
                return 1
            end

        case show
            if test -z "$bookmark_name"
                echo "Usage: bf show <bookmark_name >"
                return 1
            end

            if set -q "$var_name"
                echo "$$var_name"
            else
                echo "Bookmark '$bookmark_name' not found."
                return 1
            end

        case export
            set -l filename $argv[2]
            if test -z "$filename"
                echo "Usage: bf export <filename >"
                return 1
            end

            # Get all universal variables starting with '$PREFIX'
            set -l bookmarks (set -U -n | string match -er "^$PREFIX")
            if test (count $bookmarks) -eq 0
                echo "No bookmarks to export."
                return 0
            end

            # Clear the file before writing
            echo -n >"$filename"

            for var in $bookmarks
                # Remove the '$PREFIX' prefix to get the clean name
                set -l clean_name (string replace "$PREFIX" '' -- $var)
                # Get the path value
                set -l path $$var
                echo "$clean_name=$path" >>"$filename"
            end

            echo "Exported (count $bookmarks) bookmarks to '$filename'"

        case import
            set -l filename $argv[2]
            if test -z "$filename"
                echo "Usage: bf import <filename >"
                return 1
            end

            if not test -e "$filename"
                echo "Error: File '$filename' not found."
                return 1
            end

            set -l count 0
            while read -l line
                set -l parts (string split -m 1 "=" -- "$line")
                if test (count $parts) -eq 2
                    set -l name $parts[1]
                    set -l path $parts[2]
                    # Set the universal variable, overwriting if it exists
                    set -U "$PREFIX$name" "$path"
                    set count (math $count + 1)
                end
            end <"$filename"

            echo "Imported $count bookmarks from '$filename'."

        case '*' # Default case for no subcommand or an unknown one
            echo "Bookmark Manager"
            echo "Usage: bf <subcommand > [arguments]"
            echo ""
            echo "Subcommands:"
            echo "  save <name >        Saves the current directory as a bookmark"
            echo "  go   <name >        Goes to the directory of the specified bookmark"
            echo "  list               Lists all saved bookmarks"
            echo "  del  <name >        Deletes the specified bookmark"
            echo "  export <filename >  Export the bookmark list to file"
            echo "  import <filename >  Import bookmarks from a file"
            return 1
    end
end

Keep long-running commands in the background

Yet another annoyance with a simple fix. I often have to keep long-running processes around, like npm run dev or docker run postgres. Of course, the simple solution for this kind of thing is

nohup <command> > /dev/null 2>&1 &

If you're not familiar with it, here is what each part does:

  • &: run the command in the background,
  • > /dev/null 2>&1: suppress any output,
  • nohup: ensure the process keeps running even if the parent process (terminal session) terminates.

This works well, but it's a bit tedious to write. Terminating running commands is a pain (you have to find the right process first, then terminate it manually), and output is fully suppressed with no way to get it back. You can pipe to a log file instead, but then you need to worry about cleanup.

I built a small wrapper around this concept called devctl that essentially does this, but keeps track of processes per working directory, automatically keeps logs accessible, and makes it easy to stop the processes. Most of this 'state' is stored in XDG_RUNTIME_DIR (or /tmp when it isn't set, as on macOS), so it is not expected to survive a restart, which is useful for log cleanup and to make sure no phantom processes remain after a restart.

Usage
$ devctl start label "sleep 30"
# started label (pid 78339)
# logs: /tmp/xdg-runtime-501/devctl/4a1963e2013b8b9f/label/stdout.log

$ devctl status
# label            RUNNING   pid=78339   uptime=0m03s     sleep 30

$ devctl stop label
# stopped label
  
$ devctl status
# label            STOPPED   pid=78339   uptime=-         sleep 30

# view logs: devctl logs label
# view stderr logs: devctl logs label --stderr
# stream stderr logs: devctl logs label -f --stderr
Source
#!/usr/bin/env uv run --script
"""devctl: a tiny, daemon-free process manager for development commands."""

import argparse
import hashlib
import json
import os
import signal
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path

DEFAULT_TIMEOUT = 8


def runtime_root():
    root = os.environ.get("XDG_RUNTIME_DIR")
    if root:
        path = Path(root).expanduser()
        if not path.is_dir():
            raise ValueError(f"XDG_RUNTIME_DIR is not a directory: {path}")
        result = path / "devctl"
    else:
        # macOS has no XDG_RUNTIME_DIR by default; /tmp is close enough for logs and state.
        result = Path("/tmp") / f"devctl-{os.getuid()}"
    result.mkdir(mode=0o700, parents=True, exist_ok=True)
    return result


def workdir_key(cwd=None):
    directory = Path(cwd or Path.cwd()).expanduser().resolve()
    return hashlib.sha256(str(directory).encode()).hexdigest()[:16]


def proc_dir(name, cwd=None):
    if not name or name in {".", ".."} or "/" in name or "\0" in name:
        raise ValueError("process name must be a simple name (no slashes)")
    return runtime_root() / workdir_key(cwd) / name


def read_meta(name, cwd=None):
    path = proc_dir(name, cwd) / "process.json"
    if not path.exists():
        raise ValueError(f"unknown process: {name}")
    try:
        return json.loads(path.read_text())
    except (OSError, json.JSONDecodeError) as exc:
        raise ValueError(f"could not read metadata for {name}: {exc}")


def pid_alive(pid):
    try:
        os.kill(pid, 0)
        return True
    except ProcessLookupError:
        return False
    except PermissionError:
        return True


def start_token(pid):
    # The start time pins a PID to one process, so a recycled PID is not mistaken for ours.
    result = subprocess.run(["ps", "-p", str(pid), "-o", "lstart="],
                            capture_output=True, text=True, check=False)
    return result.stdout.strip() or None


def is_running(meta):
    # The child is launched in a new session, so its PID is also its process-group ID.
    pid = int(meta["pid"])
    if not pid_alive(pid):
        return False
    token = meta.get("start_token")
    return token is None or start_token(pid) == token


def start(args):
    cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
    if not cwd.is_dir():
        raise ValueError(f"working directory does not exist: {cwd}")
    directory = proc_dir(args.name, cwd)
    directory.mkdir(parents=True, exist_ok=True)
    meta_path = directory / "process.json"

    if meta_path.exists():
        old = json.loads(meta_path.read_text())
        if is_running(old):
            print(f"{args.name} is already running (pid {old['pid']})")
            return 1

    out = open(directory / "stdout.log", "ab", buffering=0)
    err = open(directory / "stderr.log", "ab", buffering=0)
    try:
        # shell=True intentionally allows normal shell command syntax, pipes, etc.
        child = subprocess.Popen(
            args.command,
            shell=True,
            executable="/bin/sh",
            cwd=str(cwd),
            stdin=subprocess.DEVNULL,
            stdout=out,
            stderr=err,
            start_new_session=True,
            close_fds=True,
        )
    finally:
        out.close()
        err.close()

    meta = {
        "name": args.name,
        "command": args.command,
        "cwd": str(cwd),
        "pid": child.pid,
        "start_token": start_token(child.pid),
        "started_at": time.time(),
        "health_url": args.health,
    }
    temp = meta_path.with_suffix(".json.tmp")
    temp.write_text(json.dumps(meta, indent=2) + "\n")
    temp.replace(meta_path)
    print(f"started {args.name} (pid {child.pid})")
    print(f"logs: {directory / 'stdout.log'}")
    return 0


def stop(args):
    meta = read_meta(args.name, args.cwd)
    pid = int(meta["pid"])
    if not is_running(meta):
        print(f"{args.name} is not running")
        return 0

    try:
        os.killpg(pid, signal.SIGTERM)
    except ProcessLookupError:
        print(f"{args.name} is already stopped")
        return 0

    deadline = time.monotonic() + args.timeout
    while time.monotonic() < deadline:
        if not is_running(meta):
            print(f"stopped {args.name}")
            return 0
        time.sleep(0.15)

    try:
        os.killpg(pid, signal.SIGKILL)
    except ProcessLookupError:
        pass
    print(f"stopped {args.name} (after force-kill timeout)")
    return 0


def status_one(name, cwd=None, quiet=False):
    meta = read_meta(name, cwd)
    running = is_running(meta)
    elapsed = max(0, int(time.time() - meta["started_at"])) if running else 0
    if running:
        mins, secs = divmod(elapsed, 60)
        hours, mins = divmod(mins, 60)
        uptime = f"{hours}h{mins:02}m" if hours else f"{mins}m{secs:02}s"
        state = "RUNNING"
    else:
        uptime, state = "-", "STOPPED"
    if not quiet:
        print(f"{name:<16} {state:<9} pid={meta['pid']:<7} uptime={uptime:<9} {meta['command']}")
    return running, meta


def list_processes(args):
    cwd = Path(args.cwd).expanduser().resolve() if getattr(args, "cwd", None) else Path.cwd().resolve()
    project_dir = runtime_root() / workdir_key(cwd)
    found = False
    if project_dir.exists():
        for child in sorted(project_dir.iterdir()):
            if child.is_dir() and (child / "process.json").exists():
                found = True
                try:
                    status_one(child.name, cwd)
                except ValueError as exc:
                    print(exc, file=sys.stderr)
    if not found:
        print("no managed processes for this workdir")
    return 0


def status(args):
    if args.name:
        status_one(args.name, args.cwd)
    else:
        return list_processes(args)
    return 0


def logs(args):
    directory = proc_dir(args.name, args.cwd)
    stream = "stderr.log" if args.stderr else "stdout.log"
    path = directory / stream
    if not path.exists():
        raise ValueError(f"no log file yet: {path}")
    if args.follow:
        # Delegate follow mode to tail; it handles rotation/truncation better.
        os.execvp("tail", ["tail", "-n", str(args.lines), "-f", str(path)])
    with path.open("r", errors="replace") as f:
        lines = f.readlines()
    sys.stdout.writelines(lines[-args.lines:])
    return 0


def health(args):
    cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
    project_dir = runtime_root() / workdir_key(cwd)
    names = [args.name] if args.name else [
        p.name for p in sorted(project_dir.iterdir())
        if p.is_dir() and (p / "process.json").exists()
    ] if project_dir.exists() else []
    if not names:
        print("no managed processes")
        return 0

    failed = False
    for name in names:
        meta = read_meta(name, args.cwd)
        if not is_running(meta):
            print(f"{name:<16} DOWN (process stopped)")
            failed = True
            continue
        url = meta.get("health_url")
        if not url:
            print(f"{name:<16} RUNNING (no health URL configured)")
            continue
        try:
            req = urllib.request.Request(url, headers={"User-Agent": "devctl/1.0"})
            with urllib.request.urlopen(req, timeout=args.timeout) as response:
                code = response.status
            ok = 200 <= code < 400
            print(f"{name:<16} {'OK' if ok else 'FAIL'} HTTP {code} {url}")
            failed = failed or not ok
        except Exception as exc:
            print(f"{name:<16} FAIL {url} ({exc})")
            failed = True
    return 1 if failed else 0


def remove(args):
    directory = proc_dir(args.name, args.cwd)
    meta = read_meta(args.name, args.cwd)
    if is_running(meta):
        raise ValueError(f"{args.name} is still running; stop it before removing")
    import shutil
    shutil.rmtree(directory)
    print(f"removed {args.name}")
    return 0



def start_batch(args):
    cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
    if args.name or args.command:
        raise ValueError("batch mode takes no positional name or command")
    if sys.stdin.isatty():
        raise ValueError("provide batch commands on stdin using a heredoc")
    count = failures = 0
    for number, raw in enumerate(sys.stdin, 1):
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        if ":" not in line:
            print(f"line {number}: expected 'name: command'", file=sys.stderr)
            failures += 1
            continue
        name, command = (part.strip() for part in line.split(":", 1))
        if not name or not command:
            print(f"line {number}: expected non-empty name and command", file=sys.stderr)
            failures += 1
            continue
        ns = argparse.Namespace(name=name, command=command, cwd=str(cwd), health=None)
        try:
            result = start(ns)
            if result:
                failures += 1
            else:
                count += 1
        except (ValueError, OSError) as exc:
            print(f"{name}: {exc}", file=sys.stderr)
            failures += 1
    print(f"started {count} process(es); {failures} error(s)")
    return 1 if failures else 0


def build_parser():
    parser = argparse.ArgumentParser(
        prog="devctl",
        description="Manage detached development processes without a multiplexer.",
        epilog="Runtime state and logs live under $XDG_RUNTIME_DIR/devctl (or /tmp/devctl-$UID), scoped by working directory.",
    )
    sub = parser.add_subparsers(dest="action", required=True)

    p = sub.add_parser("start", help="start a command in the background")
    p.add_argument("name", nargs="?", help="process name; omit when reading a batch from stdin")
    p.add_argument("command", nargs="?", help="command string, run by /bin/sh")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.add_argument("--health", help="HTTP URL to check, e.g. http://127.0.0.1:3000/")
    p.set_defaults(func=start)

    p = sub.add_parser("stop", help="stop a process and its process group")
    p.add_argument("name")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT, help="seconds before SIGKILL (default: 8)")
    p.set_defaults(func=stop)

    p = sub.add_parser("restart", help="restart a process using its saved command")
    p.add_argument("name")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT)
    p.set_defaults(func=restart)

    p = sub.add_parser("status", help="show one process or all processes")
    p.add_argument("name", nargs="?")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.set_defaults(func=status)

    p = sub.add_parser("list", help="list managed processes in this workdir")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.set_defaults(func=list_processes)

    p = sub.add_parser("logs", help="show logs")
    p.add_argument("name")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.add_argument("-n", "--lines", type=int, default=50)
    p.add_argument("-f", "--follow", action="store_true")
    p.add_argument("--stderr", action="store_true", help="show stderr instead of stdout")
    p.set_defaults(func=logs)

    p = sub.add_parser("health", help="check process state and configured HTTP endpoints")
    p.add_argument("name", nargs="?")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.add_argument("--timeout", type=float, default=2.5, help="HTTP timeout in seconds")
    p.set_defaults(func=health)

    p = sub.add_parser("remove", help="remove saved metadata and logs for a stopped process")
    p.add_argument("name")
    p.add_argument("--cwd", help="working directory (default: current directory)")
    p.set_defaults(func=remove)
    return parser


def restart(args):
    meta = read_meta(args.name, args.cwd)
    if is_running(meta):
        stop(argparse.Namespace(name=args.name, cwd=args.cwd, timeout=args.timeout))
    ns = argparse.Namespace(
        name=args.name,
        command=meta["command"],
        cwd=meta["cwd"],
        health=meta.get("health_url"),
    )
    return start(ns)


def main():
    parser = build_parser()
    args = parser.parse_args()
    if getattr(args, "lines", 1) < 0:
        parser.error("--lines must be non-negative")
    try:
        if args.action == "start":
            if args.name is None and args.command is None:
                return start_batch(args)
            if args.name is None or args.command is None:
                parser.error("start requires both NAME and COMMAND, or a heredoc batch")
        return args.func(args) or 0
    except (ValueError, OSError, KeyError) as exc:
        print(f"devctl: {exc}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    sys.exit(main())

If you haven't seen it before, the script above contains a uv shebang #!/usr/bin/env uv run --script. This is really useful for standalone Python scripts with dependencies defined according to PEP 723.

Quick PRs from Jujutsu

This is one of those scripts I use all the time, yet it's so simple. As you may have read, I'm a big fan of the VCS Jujutsu. At this point, it has completely replaced git for me.1 I won't give a full recap of jj here (I recommend reading my post if you're interested), but jj doesn't have branches like git; instead, it has bookmarks. Yet they fulfill a similar function and can be used with git forges like GitHub to represent PRs.

So let's say I have some work on revision xxlrk that I want to publish as a PR. With jj, I can push a revision as a branch directly, which automatically creates a bookmark for me (very useful). All I have to do is jj git push -c xxlrk (which I abbreviate as jj p -c xxlrk). So my branch is on GitHub and I have a tracked bookmark push-xxlrk matching that branch. Now I want to create a PR, and I have multiple options:

  • open GitHub in my browser and make the PR (what I used to do).
  • use the gh CLI with gh pr create (requires that I somehow communicate which branch to use, not very nice with hash-based names).
  • use jj pr.

Clearly, the latter option is the easiest, and it only requires a custom script to define the operation. The main idea is to resolve the requested bookmarks (defaulting to the nearest ancestor bookmark and main/master) and then defer to gh pr create.

Source
#!/usr/bin/env bash

_jjpr_default_from() {
    jj log --color never --revisions '::@' --no-graph \
        --template 'bookmarks ++ "\n"' --ignore-working-copy |
        tr ' ' '\n' |
        sed 's/\*.*$//' |
        grep -v -E '^(main|master)$' |
        grep -v '^$' |
        head -n 1
}

_jjpr_default_to() {
    local bookmark

    for bookmark in main master; do
        if jj log --color never --revisions "$bookmark" \
            --no-graph --template bookmarks --ignore-working-copy \
            >/dev/null 2>&1; then
            printf '%s\n' "$bookmark"
            return 0
        fi
    done

    return 0
}

_jjpr_remote_bookmark() {
    jj log --color never \
        --revisions "remote_bookmarks() & $1" \
        --no-graph --template bookmarks --ignore-working-copy |
        tr ' ' '\n' |
        sed 's/\*.*$//' |
        head -n 1
}

jjpr() {
    local from=''
    local to=''
    local existing
    local bookmark

    while (($#)); do
        case "$1" in
            --from=*)
                from=${1#--from=}
                ;;
            --from)
                if (($# < 2)); then
                    printf 'Option --from requires a bookmark.\n' >&2
                    return 2
                fi
                from=$2
                shift
                ;;
            --to=*)
                to=${1#--to=}
                ;;
            --to)
                if (($# < 2)); then
                    printf 'Option --to requires a bookmark.\n' >&2
                    return 2
                fi
                to=$2
                shift
                ;;
            -h|--help)
                printf '%s\n' \
                    'Usage: jjpr [--from BOOKMARK] [--to BOOKMARK]' \
                    '  --from BOOKMARK  Source bookmark (defaults to the first ancestor bookmark)' \
                    '  --to BOOKMARK    Target bookmark (defaults to main or master)'
                return 0
                ;;
            *)
                printf 'Usage: jjpr [--from BOOKMARK] [--to BOOKMARK]\n' >&2
                return 2
                ;;
        esac
        shift
    done

    if [[ -z $from ]]; then
        from=$(_jjpr_default_from) || return
    fi

    if [[ -z $from ]]; then
        printf 'No ancestor bookmark found. Specify --from BOOKMARK.\n' >&2
        return 1
    fi

    if [[ -z $to ]]; then
        to=$(_jjpr_default_to) || return
    fi

    if [[ -z $to ]]; then
        printf 'No main or master bookmark found. Specify --to BOOKMARK.\n' >&2
        return 1
    fi

    for bookmark in "$from" "$to"; do
        if ! jj log --color never --revisions "$bookmark" \
            --no-graph --template bookmarks --ignore-working-copy \
            >/dev/null; then
            printf 'Bookmark not found: %s\n' "$bookmark" >&2
            return 1
        fi

        jj git push --bookmark "$bookmark" || return
    done

    local remote_from
    local remote_to

    remote_from=$(_jjpr_remote_bookmark "$from") || return
    remote_to=$(_jjpr_remote_bookmark "$to") || return

    if [[ -z $remote_from ]]; then
        printf 'Bookmark is not tracked on a remote: %s\n' "$from" >&2
        return 1
    fi

    if [[ -z $remote_to ]]; then
        printf 'Bookmark is not tracked on a remote: %s\n' "$to" >&2
        return 1
    fi

    existing=$(gh pr list --state all --base "$to" --head "$from" \
        --json number --jq '.[0].number') || return

    if [[ -n $existing ]]; then
        printf 'Pull request already exists: #%s (%s -> %s)\n' \
            "$existing" "$from" "$to" >&2
        return 1
    fi

    gh pr create --base "$to" --head "$from"
}

jjpr "$@"

Keeping it all synchronized

To keep everything in sync, I use a bare git repository in my $HOME, managed through a wrapper called dotf. Here is a good overview of how to use that setup to track your dotfiles. If you follow this tutorial, you'll have a repository where you can add or remove tracked files using git --git-dir=$HOME/.cfg --work-tree=$HOME (replace .cfg with whatever you used in those steps; in my case it's .dotfiles.git). Now that's a mouthful! Of course, you can make a bash/zsh/fish/whateversh alias for it. I chose to have a wrapper script instead, because I wanted to introduce some other features:

  • dotf push automatically stages tracked files, commits with an empty message and pushes to origin.
  • dotf pull automatically stashes current changes, pulls new commits, and pops the stash on top.
  • dotf list lists tracked, untracked, and changed files in the current directory (useful given how sparse dotfiles can be). dotf watch lists the watched folders.
  • Any folder containing a tracked .dotf file (empty is fine) will auto-track every file in that folder. dotf status shows the untracked files that push will add.
Source
#!/usr/bin/env python3

import argparse
import os
from pathlib import Path
import subprocess
import sys


class Dotfiles:
    def __init__(self, args):
        self.home = Path(args.work_tree).expanduser().resolve()
        self.repo = Path(args.repo).expanduser().resolve()
        if not self.repo.is_dir():
            raise ValueError(f"Repository not found: {self.repo}")
        self.env = dict(os.environ, GIT_DIR=str(self.repo), GIT_WORK_TREE=str(self.home),
                        GIT_LITERAL_PATHSPECS="1")
        self.git("rev-parse", "--git-dir", capture=True)

    def git(self, *args, capture=False, check=True, quiet=False, input=None):
        return subprocess.run(
            ["git", *args], cwd=self.home, env=self.env, text=True, input=input,
            errors="surrogateescape", check=check,
            stdout=subprocess.PIPE if capture else None,
            stderr=subprocess.DEVNULL if quiet else None,
        )

    def files(self, *args):
        return set(filter(None, self.git(*args, capture=True).stdout.split("\0")))

    def watches(self):
        parents = {Path(path).parent.as_posix()
                   for path in self.files("ls-files", "--cached", "-z")
                   if Path(path).name == ".dotf" and (self.home / path).is_file()}
        watched = []
        for parent in sorted(parents - {"."}):
            path = (self.home / parent).resolve()
            if (path.is_relative_to(self.home) and path != self.home
                    and not path.is_relative_to(self.repo)
                    and not self.repo.is_relative_to(path)):
                watched.append(parent)
        return watched

    def relative(self, path):
        path = Path(path).expanduser().resolve()
        try:
            return path.relative_to(self.home).as_posix()
        except ValueError:
            raise ValueError(f"Path is outside the work tree: {path}") from None

    def watch(self, args):
        self.show("Folders with tracked .dotf markers (auto-add recursively on push)", self.watches())
        print("The home directory and Git repository are excluded. Git ignore rules apply.")

    @staticmethod
    def show(title, files):
        print(f"{title}:")
        for path in sorted(files):
            print(f"  {path!r}" if not path.isprintable() else f"  {path}")
        if not files:
            print("  (none)")

    def status(self, args):
        self.git("status", "--untracked-files=no")
        watched = self.watches()
        if watched:
            self.show("Watched files that push will add",
                      self.files("ls-files", "--others", "--exclude-standard", "-z",
                                 "--", *watched))

    def list(self, args):
        directory = Path(args.directory).expanduser().resolve()
        scope = self.relative(directory)
        if not directory.is_dir():
            raise ValueError(f"Not a directory: {directory}")
        tracked = self.files("ls-files", "--cached", "-z", "--", scope)
        untracked_args = ["ls-files", "--others", "--exclude-standard", "-z"]
        if self.repo.is_relative_to(self.home):
            untracked_args.append(f"--exclude=/{self.repo.relative_to(self.home).as_posix()}/")
        untracked = self.files(*untracked_args, "--", scope)
        changed = self.files("diff", "--name-only", "--no-renames", "-z", "--", scope)
        changed |= self.files("diff", "--cached", "--name-only", "--no-renames", "-z",
                              "--", scope)
        watched = self.watches()
        candidates = {path for path in untracked
                      if any(path.startswith(folder + "/") for folder in watched)}
        display = lambda files: {os.path.relpath(self.home / path, directory) for path in files}
        self.show("Tracked", display(tracked))
        self.show("Untracked (not ignored)", display(untracked - candidates))
        self.show("Untracked in watched folders (auto-add on push)", display(candidates))
        self.show("Uncommitted changes (push will commit these)", display(changed))
        upstream = self.git("rev-parse", "--verify", "@{upstream}", capture=True,
                            check=False, quiet=True)
        if upstream.returncode == 0:
            pending = self.files("diff", "--name-only", "--no-renames", "-z",
                                 "@{upstream}...HEAD", "--", scope)
            self.show("Committed, not pushed (local upstream reference)", display(pending))
        else:
            print("Committed, not pushed: unavailable (no upstream configured).")

    def push(self, args):
        watched = self.watches()
        if self.files("ls-files", "--unmerged", "-z"):
            raise ValueError("Resolve Git conflicts before you push.")
        self.git("add", "--update", "--", ".")
        if watched:
            new = self.files("ls-files", "--others", "--exclude-standard", "-z", "--", *watched)
            if new:
                self.git("add", "--pathspec-from-file=-", "--pathspec-file-nul",
                         input="".join(path + "\0" for path in sorted(new)))
        changed = self.git("diff", "--cached", "--quiet", check=False)
        if changed.returncode == 1:
            self.git("commit", "--allow-empty-message", "-m", args.message)
        elif changed.returncode != 0:
            raise subprocess.CalledProcessError(changed.returncode, changed.args)
        else:
            print("No changes to commit.", flush=True)
        self.git("push")

    def pull(self, args):
        dirty = self.git("status", "--porcelain", "--untracked-files=no", capture=True).stdout
        stash = None
        if dirty:
            self.git("stash", "push", "-m", "dotf pull")
            stash = self.git("rev-parse", "refs/stash", capture=True).stdout.strip()
        try:
            pulled = self.git("-c", "rebase.autoStash=false", "-c", "merge.autoStash=false",
                              "pull", "--ff-only", check=False)
        finally:
            if stash:
                restored = self.git("stash", "apply", "--index", stash, check=False)
                if restored.returncode != 0:
                    print(f"dotf: Restore failed. Your changes remain in stash {stash}.",
                          file=sys.stderr)
                    print("Resolve conflicts before you push. Do not apply the stash twice.",
                          file=sys.stderr)
                    raise subprocess.CalledProcessError(restored.returncode, restored.args)
                latest = self.git("rev-parse", "refs/stash", capture=True, check=False)
                if latest.stdout.strip() == stash:
                    self.git("stash", "drop", "stash@{0}")
                else:
                    print(f"Changes restored. Stash {stash} remains for safety.")
        if pulled.returncode != 0:
            self.divergence_hint()
            raise subprocess.CalledProcessError(pulled.returncode, pulled.args)

    def divergence_hint(self):
        counts = {}
        for key, rev in (("ahead", "@{u}..HEAD"), ("behind", "HEAD..@{u}")):
            result = self.git("rev-list", "--count", rev, capture=True, check=False)
            if result.returncode != 0:
                return
            counts[key] = result.stdout.strip()
        if counts["behind"] == "0":
            return
        upstream = self.git("rev-parse", "--abbrev-ref", "@{u}", capture=True, check=False)
        print(f"dotf: {counts['ahead']} commit(s) ahead and {counts['behind']} behind "
              f"{upstream.stdout.strip()}.", file=sys.stderr)
        print("dotf: Replay your commits with 'dotf rebase @{u}'.", file=sys.stderr)


def main():
    options = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
    options.add_argument("--repo", default="~/.dotfiles.git", help="Git repository (default: ~/.dotfiles.git)")
    options.add_argument("--work-tree", default="~", help="Work tree (default: home directory)")
    parser = argparse.ArgumentParser(
        parents=[options], allow_abbrev=False,
        description="Manage dotfiles with a Git repository and your home directory as its work tree.",
        epilog="Other commands pass directly to Git. Push auto-adds files recursively in folders with a tracked .dotf marker, except the home directory. Git ignore rules apply.",
    )
    commands = parser.add_subparsers(dest="command", required=True)
    commands.add_parser("pull", help="Stash tracked edits, pull --ff-only, then restore edits and staging")
    push = commands.add_parser("push", help="Commit tracked edits and watched files, then push")
    push.add_argument("-m", "--message", default="", help="Commit message (default: empty)")
    listing = commands.add_parser("list", help="List files and pending changes in the current directory")
    listing.add_argument("directory", nargs="?", default=".")
    commands.add_parser("status", help="Show Git status and new files in watched folders")
    watch = commands.add_parser("watch", help="Show folders with tracked .dotf markers")
    watch.add_argument("action", nargs="?", choices=["list"], help="List folders with tracked .dotf markers")
    options.add_argument("git_args", nargs=argparse.REMAINDER)
    try:
        args, unknown = options.parse_known_args()
        if not unknown and args.git_args and args.git_args[0] not in commands.choices:
            repo = Path(args.repo).expanduser().resolve()
            home = Path(args.work_tree).expanduser().resolve()
            os.execvp("git", ["git", f"--git-dir={repo}", f"--work-tree={home}", *args.git_args])
        args = parser.parse_args()
        getattr(Dotfiles(args), args.command)(args)
    except (ValueError, OSError) as error:
        print(f"dotf: {error}", file=sys.stderr)
        return 1
    except subprocess.CalledProcessError as error:
        return error.returncode
    except KeyboardInterrupt:
        print("dotf: Interrupted.", file=sys.stderr)
        return 130
    return 0


if __name__ == "__main__":
    sys.exit(main())

Conclusion

These scripts might seem random, but they really reflect my everyday workflows, and I can't overstate how much I've come to rely on each and every one of them. If you have cool ones you want to share, or if you have any notes or suggestions, you can reach out to me; I'd love to hear about them.


  1. Well, insofar as one can, given that jj currently relies on git as its backend. ↩︎