An ode to unsung heroes
Introduction
I always write about large projects, shiny things I feel particularly proud of. But once in a while, it's nice to recognize how much of that work relies on load-bearing scripts, written quickly to get something else done, and improved over time. I've reached the point where I sometimes take these little scripts for granted on other systems, and even go so far as to ensure they're present on my work machine and my VPS. So, without further ado, let's take a little tour of these useful scripts. And if you have any recommendations on how to improve these further, please let me know.
My local secrets store
The most useful script I have is called key and solves a simple need: how do I keep track of all my API keys and various non-critical secrets? The answer is essentially a wrapper around cryptographic stores on either Linux (secret-tool or pass if available) or Darwin (using a dedicated keychain via security).
The threat model here is to keep a compromised application from snooping on .env files or any other plain-text storage of my credentials and API keys. Access is not restricted beyond the defaults of the underlying tools: any process can just call key. For most use cases, this is all I need, and infinitely better than having sensitive data written anywhere on disk unencrypted.
$ key set test
# Value for test: <stdin = "value">
# Saved 'test' in Keychain.
$ key list
# test
$ key get test
# value#!/usr/bin/env bash
set -euo pipefail
usage() {
echo "Usage: key {get|set|del} <name>" >&2
echo " <value> | key set <name>" >&2
echo " key list " >&2
exit 2
}
[[ $# -ge 1 ]] || usage
command=$1
case "$command" in
list)
[[ $# -eq 1 ]] || usage
name=""
;;
get|set|del)
[[ $# -eq 2 ]] || usage
name=$2
;;
*)
usage
;;
esac
read_value() {
if [[ -t 0 ]]; then
IFS= read -r -s -p "Value for $name: " value
echo
else
# Read everything so multi-line values (PEM, JSON) survive a pipe.
value=$(cat)
fi
# strip one trailing carriage return left by clipboard tools
value=${value%$'\r'}
[[ -n "$value" ]] || { echo "No value provided." >&2; exit 1; }
}
list_keys() {
security dump-keychain "$1" | awk -v user="$USER" '
/class: "genp"/ {
if (in_item && account == user && service != "")
print service
in_item = 1
account = service = ""
next
}
in_item && /"acct"<blob>="/ {
line = $0
sub(/^.*"acct"<blob>="/, "", line)
sub(/".*$/, "", line)
account = line
}
in_item && /"svce"<blob>="/ {
line = $0
sub(/^.*"svce"<blob>="/, "", line)
sub(/".*$/, "", line)
service = line
}
END {
if (in_item && account == user && service != "")
print service
}
'
}
read_secret() {
local info hex
if ! info=$(security find-generic-password -a "$USER" -s "$1" -g "$2" 2>&1 >/dev/null); then
printf '%s\n' "$info" >&2
return 1
fi
case "$info" in
"password: 0x"*)
# `security` prints non-printable values as hex; decode them back.
hex=${info#password: 0x}
hex=${hex%% *}
printf '%b' "$(sed 's/../\\x&/g' <<<"$hex")"
;;
*)
security find-generic-password -a "$USER" -s "$1" -w "$2"
;;
esac
}
if [[ "$OSTYPE" == darwin* ]]; then
keychain="${KEY_KEYCHAIN:-$HOME/Library/Keychains/keys.keychain-db}"
login_keychain="$HOME/Library/Keychains/login.keychain-db"
# A dedicated keychain keeps `key list` fast; its password is stored in the login keychain.
if [[ ! -f $keychain ]]; then
keychain_password=$(openssl rand -hex 32)
security create-keychain -p "$keychain_password" "$keychain"
security add-generic-password -U -a "$USER" -s key-keychain-password \
-w "$keychain_password" "$login_keychain"
fi
if ! keychain_password=$(security find-generic-password -a "$USER" \
-s key-keychain-password -w "$login_keychain" 2>/dev/null); then
echo "No password stored for $keychain; remove the keychain to recreate it." >&2
exit 1
fi
security unlock-keychain -p "$keychain_password" "$keychain"
case "$command" in
get)
read_secret "$name" "$keychain"
;;
set)
read_value
security add-generic-password -U -a "$USER" -s "$name" \
-w "$value" "$keychain"
echo "Saved '$name' in Keychain."
;;
del)
if ! security find-generic-password -a "$USER" -s "$name" \
"$keychain" >/dev/null 2>&1; then
echo "No key named '$name'." >&2
exit 1
fi
security delete-generic-password -a "$USER" -s "$name" "$keychain" \
>/dev/null 2>&1 || {
echo "Failed to delete '$name'." >&2
exit 1
}
echo "Deleted '$name' from Keychain."
;;
list)
list_keys "$keychain"
;;
*)
usage
;;
esac
elif [[ "$OSTYPE" == linux* ]]; then
if command -v secret-tool >/dev/null 2>&1; then
case "$command" in
get) secret-tool lookup service key name "$name" ;;
set)
read_value
printf '%s' "$value" | secret-tool store --label="key $name" service key name "$name"
echo "Saved '$name' in keyring."
;;
del) secret-tool clear service key name "$name" && echo "Deleted '$name' from keyring." ;;
list) echo "Function not supported on Linux yet" ;;
*) usage ;;
esac
elif command -v pass >/dev/null 2>&1; then
case "$command" in
get) pass show "$name" ;;
set)
read_value
printf '%s\n' "$value" | pass insert -m "$name"
echo "Saved '$name' in pass."
;;
del)
store="${PASSWORD_STORE_DIR:-$HOME/.password-store}"
if [[ ! -f $store/$name.gpg ]]; then
echo "No key named '$name'." >&2
exit 1
fi
pass rm -f "$name" >/dev/null
echo "Deleted '$name' from pass."
;;
list)
store="${PASSWORD_STORE_DIR:-$HOME/.password-store}"
if [[ -d $store ]]; then
(cd "$store" && find . -type f -name '*.gpg' |
sed -e 's|^\./||' -e 's|\.gpg$||' | sort)
fi
;;
*) usage ;;
esac
else
echo "Linux requires 'secret-tool' or 'pass'." >&2
exit 1
fi
else
echo "Unsupported operating system: $OSTYPE" >&2
exit 1
fiHomemade fish bookmarks
Even though I use zoxide instead of cd, I still like to have bookmarks for folders I keep switching between. This is the kind of script you write yourself, because existing tools never quite fit. So here is mine, as a fish function, using universal variables as the backend.
cd /some/path
bf save "label"
cd ..
bf go label#!/usr/bin/env fish
#
# Bookmark manager using universal variables in fish
function bf -d "Bookmark manager for directories"
set -l PREFIX bkm_
# The subcommand is the first argument (e.g., 'save', 'go')
set -l subcommand $argv[1]
# The name of the bookmark is the second argument
set -l bookmark_name $argv[2]
# The full universal variable name with our prefix
set -l var_name "$PREFIX$bookmark_name"
switch $subcommand
case save
if not string match -qr '^[A-Za-z0-9._-]+$' -- "$bookmark_name"
echo "Invalid bookmark name: $bookmark_name" >&2
return 1
end
set -U "$var_name" (pwd)
echo "Saved bookmark '$bookmark_name'"
case go
if test -z "$bookmark_name"
echo "Usage: bf go <bookmark_name>"
# If no name provided, list available bookmarks
bf list
return 1
end
if set -q "$var_name"
cd "$$var_name"
else
echo "Bookmark '$bookmark_name' not found."
return 1
end
case list
set -l bookmarks (set -U -n | string match -er "^$PREFIX" | string replace $PREFIX '')
if test (count $bookmarks) -eq 0
echo "No bookmarks set."
else
echo "Available bookmarks:"
for name in $bookmarks
set -l var "$PREFIX$name"
echo "- $name: $$var"
end
end
case del
if test -z "$bookmark_name"
echo "Usage: bf del <bookmark_name >"
return 1
end
if set -q "$var_name"
set -e "$var_name"
echo "Deleted bookmark '$bookmark_name'."
else
echo "Bookmark '$bookmark_name' not found."
return 1
end
case show
if test -z "$bookmark_name"
echo "Usage: bf show <bookmark_name >"
return 1
end
if set -q "$var_name"
echo "$$var_name"
else
echo "Bookmark '$bookmark_name' not found."
return 1
end
case export
set -l filename $argv[2]
if test -z "$filename"
echo "Usage: bf export <filename >"
return 1
end
# Get all universal variables starting with '$PREFIX'
set -l bookmarks (set -U -n | string match -er "^$PREFIX")
if test (count $bookmarks) -eq 0
echo "No bookmarks to export."
return 0
end
# Clear the file before writing
echo -n >"$filename"
for var in $bookmarks
# Remove the '$PREFIX' prefix to get the clean name
set -l clean_name (string replace "$PREFIX" '' -- $var)
# Get the path value
set -l path $$var
echo "$clean_name=$path" >>"$filename"
end
echo "Exported (count $bookmarks) bookmarks to '$filename'"
case import
set -l filename $argv[2]
if test -z "$filename"
echo "Usage: bf import <filename >"
return 1
end
if not test -e "$filename"
echo "Error: File '$filename' not found."
return 1
end
set -l count 0
while read -l line
set -l parts (string split -m 1 "=" -- "$line")
if test (count $parts) -eq 2
set -l name $parts[1]
set -l path $parts[2]
# Set the universal variable, overwriting if it exists
set -U "$PREFIX$name" "$path"
set count (math $count + 1)
end
end <"$filename"
echo "Imported $count bookmarks from '$filename'."
case '*' # Default case for no subcommand or an unknown one
echo "Bookmark Manager"
echo "Usage: bf <subcommand > [arguments]"
echo ""
echo "Subcommands:"
echo " save <name > Saves the current directory as a bookmark"
echo " go <name > Goes to the directory of the specified bookmark"
echo " list Lists all saved bookmarks"
echo " del <name > Deletes the specified bookmark"
echo " export <filename > Export the bookmark list to file"
echo " import <filename > Import bookmarks from a file"
return 1
end
endKeep long-running commands in the background
Yet another annoyance with a simple fix. I often have to keep long-running processes around, like npm run dev or docker run postgres. Of course, the simple solution for this kind of thing is
nohup <command> > /dev/null 2>&1 &If you're not familiar with it, here is what each part does:
&: run the command in the background,> /dev/null 2>&1: suppress any output,nohup: ensure the process keeps running even if the parent process (terminal session) terminates.
This works well, but it's a bit tedious to write. Terminating running commands is a pain (you have to find the right process first, then terminate it manually), and output is fully suppressed with no way to get it back. You can pipe to a log file instead, but then you need to worry about cleanup.
I built a small wrapper around this concept called devctl that essentially does this, but keeps track of processes per working directory, automatically keeps logs accessible, and makes it easy to stop the processes. Most of this 'state' is stored in XDG_RUNTIME_DIR (or /tmp when it isn't set, as on macOS), so it is not expected to survive a restart, which is useful for log cleanup and to make sure no phantom processes remain after a restart.
$ devctl start label "sleep 30"
# started label (pid 78339)
# logs: /tmp/xdg-runtime-501/devctl/4a1963e2013b8b9f/label/stdout.log
$ devctl status
# label RUNNING pid=78339 uptime=0m03s sleep 30
$ devctl stop label
# stopped label
$ devctl status
# label STOPPED pid=78339 uptime=- sleep 30
# view logs: devctl logs label
# view stderr logs: devctl logs label --stderr
# stream stderr logs: devctl logs label -f --stderr#!/usr/bin/env uv run --script
"""devctl: a tiny, daemon-free process manager for development commands."""
import argparse
import hashlib
import json
import os
import signal
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
DEFAULT_TIMEOUT = 8
def runtime_root():
root = os.environ.get("XDG_RUNTIME_DIR")
if root:
path = Path(root).expanduser()
if not path.is_dir():
raise ValueError(f"XDG_RUNTIME_DIR is not a directory: {path}")
result = path / "devctl"
else:
# macOS has no XDG_RUNTIME_DIR by default; /tmp is close enough for logs and state.
result = Path("/tmp") / f"devctl-{os.getuid()}"
result.mkdir(mode=0o700, parents=True, exist_ok=True)
return result
def workdir_key(cwd=None):
directory = Path(cwd or Path.cwd()).expanduser().resolve()
return hashlib.sha256(str(directory).encode()).hexdigest()[:16]
def proc_dir(name, cwd=None):
if not name or name in {".", ".."} or "/" in name or "\0" in name:
raise ValueError("process name must be a simple name (no slashes)")
return runtime_root() / workdir_key(cwd) / name
def read_meta(name, cwd=None):
path = proc_dir(name, cwd) / "process.json"
if not path.exists():
raise ValueError(f"unknown process: {name}")
try:
return json.loads(path.read_text())
except (OSError, json.JSONDecodeError) as exc:
raise ValueError(f"could not read metadata for {name}: {exc}")
def pid_alive(pid):
try:
os.kill(pid, 0)
return True
except ProcessLookupError:
return False
except PermissionError:
return True
def start_token(pid):
# The start time pins a PID to one process, so a recycled PID is not mistaken for ours.
result = subprocess.run(["ps", "-p", str(pid), "-o", "lstart="],
capture_output=True, text=True, check=False)
return result.stdout.strip() or None
def is_running(meta):
# The child is launched in a new session, so its PID is also its process-group ID.
pid = int(meta["pid"])
if not pid_alive(pid):
return False
token = meta.get("start_token")
return token is None or start_token(pid) == token
def start(args):
cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
if not cwd.is_dir():
raise ValueError(f"working directory does not exist: {cwd}")
directory = proc_dir(args.name, cwd)
directory.mkdir(parents=True, exist_ok=True)
meta_path = directory / "process.json"
if meta_path.exists():
old = json.loads(meta_path.read_text())
if is_running(old):
print(f"{args.name} is already running (pid {old['pid']})")
return 1
out = open(directory / "stdout.log", "ab", buffering=0)
err = open(directory / "stderr.log", "ab", buffering=0)
try:
# shell=True intentionally allows normal shell command syntax, pipes, etc.
child = subprocess.Popen(
args.command,
shell=True,
executable="/bin/sh",
cwd=str(cwd),
stdin=subprocess.DEVNULL,
stdout=out,
stderr=err,
start_new_session=True,
close_fds=True,
)
finally:
out.close()
err.close()
meta = {
"name": args.name,
"command": args.command,
"cwd": str(cwd),
"pid": child.pid,
"start_token": start_token(child.pid),
"started_at": time.time(),
"health_url": args.health,
}
temp = meta_path.with_suffix(".json.tmp")
temp.write_text(json.dumps(meta, indent=2) + "\n")
temp.replace(meta_path)
print(f"started {args.name} (pid {child.pid})")
print(f"logs: {directory / 'stdout.log'}")
return 0
def stop(args):
meta = read_meta(args.name, args.cwd)
pid = int(meta["pid"])
if not is_running(meta):
print(f"{args.name} is not running")
return 0
try:
os.killpg(pid, signal.SIGTERM)
except ProcessLookupError:
print(f"{args.name} is already stopped")
return 0
deadline = time.monotonic() + args.timeout
while time.monotonic() < deadline:
if not is_running(meta):
print(f"stopped {args.name}")
return 0
time.sleep(0.15)
try:
os.killpg(pid, signal.SIGKILL)
except ProcessLookupError:
pass
print(f"stopped {args.name} (after force-kill timeout)")
return 0
def status_one(name, cwd=None, quiet=False):
meta = read_meta(name, cwd)
running = is_running(meta)
elapsed = max(0, int(time.time() - meta["started_at"])) if running else 0
if running:
mins, secs = divmod(elapsed, 60)
hours, mins = divmod(mins, 60)
uptime = f"{hours}h{mins:02}m" if hours else f"{mins}m{secs:02}s"
state = "RUNNING"
else:
uptime, state = "-", "STOPPED"
if not quiet:
print(f"{name:<16} {state:<9} pid={meta['pid']:<7} uptime={uptime:<9} {meta['command']}")
return running, meta
def list_processes(args):
cwd = Path(args.cwd).expanduser().resolve() if getattr(args, "cwd", None) else Path.cwd().resolve()
project_dir = runtime_root() / workdir_key(cwd)
found = False
if project_dir.exists():
for child in sorted(project_dir.iterdir()):
if child.is_dir() and (child / "process.json").exists():
found = True
try:
status_one(child.name, cwd)
except ValueError as exc:
print(exc, file=sys.stderr)
if not found:
print("no managed processes for this workdir")
return 0
def status(args):
if args.name:
status_one(args.name, args.cwd)
else:
return list_processes(args)
return 0
def logs(args):
directory = proc_dir(args.name, args.cwd)
stream = "stderr.log" if args.stderr else "stdout.log"
path = directory / stream
if not path.exists():
raise ValueError(f"no log file yet: {path}")
if args.follow:
# Delegate follow mode to tail; it handles rotation/truncation better.
os.execvp("tail", ["tail", "-n", str(args.lines), "-f", str(path)])
with path.open("r", errors="replace") as f:
lines = f.readlines()
sys.stdout.writelines(lines[-args.lines:])
return 0
def health(args):
cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
project_dir = runtime_root() / workdir_key(cwd)
names = [args.name] if args.name else [
p.name for p in sorted(project_dir.iterdir())
if p.is_dir() and (p / "process.json").exists()
] if project_dir.exists() else []
if not names:
print("no managed processes")
return 0
failed = False
for name in names:
meta = read_meta(name, args.cwd)
if not is_running(meta):
print(f"{name:<16} DOWN (process stopped)")
failed = True
continue
url = meta.get("health_url")
if not url:
print(f"{name:<16} RUNNING (no health URL configured)")
continue
try:
req = urllib.request.Request(url, headers={"User-Agent": "devctl/1.0"})
with urllib.request.urlopen(req, timeout=args.timeout) as response:
code = response.status
ok = 200 <= code < 400
print(f"{name:<16} {'OK' if ok else 'FAIL'} HTTP {code} {url}")
failed = failed or not ok
except Exception as exc:
print(f"{name:<16} FAIL {url} ({exc})")
failed = True
return 1 if failed else 0
def remove(args):
directory = proc_dir(args.name, args.cwd)
meta = read_meta(args.name, args.cwd)
if is_running(meta):
raise ValueError(f"{args.name} is still running; stop it before removing")
import shutil
shutil.rmtree(directory)
print(f"removed {args.name}")
return 0
def start_batch(args):
cwd = Path(args.cwd).expanduser().resolve() if args.cwd else Path.cwd().resolve()
if args.name or args.command:
raise ValueError("batch mode takes no positional name or command")
if sys.stdin.isatty():
raise ValueError("provide batch commands on stdin using a heredoc")
count = failures = 0
for number, raw in enumerate(sys.stdin, 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
if ":" not in line:
print(f"line {number}: expected 'name: command'", file=sys.stderr)
failures += 1
continue
name, command = (part.strip() for part in line.split(":", 1))
if not name or not command:
print(f"line {number}: expected non-empty name and command", file=sys.stderr)
failures += 1
continue
ns = argparse.Namespace(name=name, command=command, cwd=str(cwd), health=None)
try:
result = start(ns)
if result:
failures += 1
else:
count += 1
except (ValueError, OSError) as exc:
print(f"{name}: {exc}", file=sys.stderr)
failures += 1
print(f"started {count} process(es); {failures} error(s)")
return 1 if failures else 0
def build_parser():
parser = argparse.ArgumentParser(
prog="devctl",
description="Manage detached development processes without a multiplexer.",
epilog="Runtime state and logs live under $XDG_RUNTIME_DIR/devctl (or /tmp/devctl-$UID), scoped by working directory.",
)
sub = parser.add_subparsers(dest="action", required=True)
p = sub.add_parser("start", help="start a command in the background")
p.add_argument("name", nargs="?", help="process name; omit when reading a batch from stdin")
p.add_argument("command", nargs="?", help="command string, run by /bin/sh")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.add_argument("--health", help="HTTP URL to check, e.g. http://127.0.0.1:3000/")
p.set_defaults(func=start)
p = sub.add_parser("stop", help="stop a process and its process group")
p.add_argument("name")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT, help="seconds before SIGKILL (default: 8)")
p.set_defaults(func=stop)
p = sub.add_parser("restart", help="restart a process using its saved command")
p.add_argument("name")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT)
p.set_defaults(func=restart)
p = sub.add_parser("status", help="show one process or all processes")
p.add_argument("name", nargs="?")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.set_defaults(func=status)
p = sub.add_parser("list", help="list managed processes in this workdir")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.set_defaults(func=list_processes)
p = sub.add_parser("logs", help="show logs")
p.add_argument("name")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.add_argument("-n", "--lines", type=int, default=50)
p.add_argument("-f", "--follow", action="store_true")
p.add_argument("--stderr", action="store_true", help="show stderr instead of stdout")
p.set_defaults(func=logs)
p = sub.add_parser("health", help="check process state and configured HTTP endpoints")
p.add_argument("name", nargs="?")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.add_argument("--timeout", type=float, default=2.5, help="HTTP timeout in seconds")
p.set_defaults(func=health)
p = sub.add_parser("remove", help="remove saved metadata and logs for a stopped process")
p.add_argument("name")
p.add_argument("--cwd", help="working directory (default: current directory)")
p.set_defaults(func=remove)
return parser
def restart(args):
meta = read_meta(args.name, args.cwd)
if is_running(meta):
stop(argparse.Namespace(name=args.name, cwd=args.cwd, timeout=args.timeout))
ns = argparse.Namespace(
name=args.name,
command=meta["command"],
cwd=meta["cwd"],
health=meta.get("health_url"),
)
return start(ns)
def main():
parser = build_parser()
args = parser.parse_args()
if getattr(args, "lines", 1) < 0:
parser.error("--lines must be non-negative")
try:
if args.action == "start":
if args.name is None and args.command is None:
return start_batch(args)
if args.name is None or args.command is None:
parser.error("start requires both NAME and COMMAND, or a heredoc batch")
return args.func(args) or 0
except (ValueError, OSError, KeyError) as exc:
print(f"devctl: {exc}", file=sys.stderr)
return 1
if __name__ == "__main__":
sys.exit(main())If you haven't seen it before, the script above contains a uv shebang #!/usr/bin/env uv run --script. This is really useful for standalone Python scripts with dependencies defined according to PEP 723.
Quick PRs from Jujutsu
This is one of those scripts I use all the time, yet it's so simple. As you may have read, I'm a big fan of the VCS Jujutsu. At this point, it has completely replaced git for me.1 I won't give a full recap of jj here (I recommend reading my post if you're interested), but jj doesn't have branches like git; instead, it has bookmarks. Yet they fulfill a similar function and can be used with git forges like GitHub to represent PRs.
So let's say I have some work on revision xxlrk that I want to publish as a PR. With jj, I can push a revision as a branch directly, which automatically creates a bookmark for me (very useful). All I have to do is jj git push -c xxlrk (which I abbreviate as jj p -c xxlrk). So my branch is on GitHub and I have a tracked bookmark push-xxlrk matching that branch. Now I want to create a PR, and I have multiple options:
- open GitHub in my browser and make the PR (what I used to do).
- use the
ghCLI withgh pr create(requires that I somehow communicate which branch to use, not very nice with hash-based names). - use
jj pr.
Clearly, the latter option is the easiest, and it only requires a custom script to define the operation. The main idea is to resolve the requested bookmarks (defaulting to the nearest ancestor bookmark and main/master) and then defer to gh pr create.
#!/usr/bin/env bash
_jjpr_default_from() {
jj log --color never --revisions '::@' --no-graph \
--template 'bookmarks ++ "\n"' --ignore-working-copy |
tr ' ' '\n' |
sed 's/\*.*$//' |
grep -v -E '^(main|master)$' |
grep -v '^$' |
head -n 1
}
_jjpr_default_to() {
local bookmark
for bookmark in main master; do
if jj log --color never --revisions "$bookmark" \
--no-graph --template bookmarks --ignore-working-copy \
>/dev/null 2>&1; then
printf '%s\n' "$bookmark"
return 0
fi
done
return 0
}
_jjpr_remote_bookmark() {
jj log --color never \
--revisions "remote_bookmarks() & $1" \
--no-graph --template bookmarks --ignore-working-copy |
tr ' ' '\n' |
sed 's/\*.*$//' |
head -n 1
}
jjpr() {
local from=''
local to=''
local existing
local bookmark
while (($#)); do
case "$1" in
--from=*)
from=${1#--from=}
;;
--from)
if (($# < 2)); then
printf 'Option --from requires a bookmark.\n' >&2
return 2
fi
from=$2
shift
;;
--to=*)
to=${1#--to=}
;;
--to)
if (($# < 2)); then
printf 'Option --to requires a bookmark.\n' >&2
return 2
fi
to=$2
shift
;;
-h|--help)
printf '%s\n' \
'Usage: jjpr [--from BOOKMARK] [--to BOOKMARK]' \
' --from BOOKMARK Source bookmark (defaults to the first ancestor bookmark)' \
' --to BOOKMARK Target bookmark (defaults to main or master)'
return 0
;;
*)
printf 'Usage: jjpr [--from BOOKMARK] [--to BOOKMARK]\n' >&2
return 2
;;
esac
shift
done
if [[ -z $from ]]; then
from=$(_jjpr_default_from) || return
fi
if [[ -z $from ]]; then
printf 'No ancestor bookmark found. Specify --from BOOKMARK.\n' >&2
return 1
fi
if [[ -z $to ]]; then
to=$(_jjpr_default_to) || return
fi
if [[ -z $to ]]; then
printf 'No main or master bookmark found. Specify --to BOOKMARK.\n' >&2
return 1
fi
for bookmark in "$from" "$to"; do
if ! jj log --color never --revisions "$bookmark" \
--no-graph --template bookmarks --ignore-working-copy \
>/dev/null; then
printf 'Bookmark not found: %s\n' "$bookmark" >&2
return 1
fi
jj git push --bookmark "$bookmark" || return
done
local remote_from
local remote_to
remote_from=$(_jjpr_remote_bookmark "$from") || return
remote_to=$(_jjpr_remote_bookmark "$to") || return
if [[ -z $remote_from ]]; then
printf 'Bookmark is not tracked on a remote: %s\n' "$from" >&2
return 1
fi
if [[ -z $remote_to ]]; then
printf 'Bookmark is not tracked on a remote: %s\n' "$to" >&2
return 1
fi
existing=$(gh pr list --state all --base "$to" --head "$from" \
--json number --jq '.[0].number') || return
if [[ -n $existing ]]; then
printf 'Pull request already exists: #%s (%s -> %s)\n' \
"$existing" "$from" "$to" >&2
return 1
fi
gh pr create --base "$to" --head "$from"
}
jjpr "$@"Keeping it all synchronized
To keep everything in sync, I use a bare git repository in my $HOME, managed through a wrapper called dotf. Here is a good overview of how to use that setup to track your dotfiles. If you follow this tutorial, you'll have a repository where you can add or remove tracked files using git --git-dir=$HOME/.cfg --work-tree=$HOME (replace .cfg with whatever you used in those steps; in my case it's .dotfiles.git). Now that's a mouthful! Of course, you can make a bash/zsh/fish/whateversh alias for it. I chose to have a wrapper script instead, because I wanted to introduce some other features:
dotf pushautomatically stages tracked files, commits with an empty message and pushes to origin.dotf pullautomatically stashes current changes, pulls new commits, and pops the stash on top.dotf listlists tracked, untracked, and changed files in the current directory (useful given how sparse dotfiles can be).dotf watchlists the watched folders.- Any folder containing a tracked
.dotffile (empty is fine) will auto-track every file in that folder.dotf statusshows the untracked files thatpushwill add.
#!/usr/bin/env python3
import argparse
import os
from pathlib import Path
import subprocess
import sys
class Dotfiles:
def __init__(self, args):
self.home = Path(args.work_tree).expanduser().resolve()
self.repo = Path(args.repo).expanduser().resolve()
if not self.repo.is_dir():
raise ValueError(f"Repository not found: {self.repo}")
self.env = dict(os.environ, GIT_DIR=str(self.repo), GIT_WORK_TREE=str(self.home),
GIT_LITERAL_PATHSPECS="1")
self.git("rev-parse", "--git-dir", capture=True)
def git(self, *args, capture=False, check=True, quiet=False, input=None):
return subprocess.run(
["git", *args], cwd=self.home, env=self.env, text=True, input=input,
errors="surrogateescape", check=check,
stdout=subprocess.PIPE if capture else None,
stderr=subprocess.DEVNULL if quiet else None,
)
def files(self, *args):
return set(filter(None, self.git(*args, capture=True).stdout.split("\0")))
def watches(self):
parents = {Path(path).parent.as_posix()
for path in self.files("ls-files", "--cached", "-z")
if Path(path).name == ".dotf" and (self.home / path).is_file()}
watched = []
for parent in sorted(parents - {"."}):
path = (self.home / parent).resolve()
if (path.is_relative_to(self.home) and path != self.home
and not path.is_relative_to(self.repo)
and not self.repo.is_relative_to(path)):
watched.append(parent)
return watched
def relative(self, path):
path = Path(path).expanduser().resolve()
try:
return path.relative_to(self.home).as_posix()
except ValueError:
raise ValueError(f"Path is outside the work tree: {path}") from None
def watch(self, args):
self.show("Folders with tracked .dotf markers (auto-add recursively on push)", self.watches())
print("The home directory and Git repository are excluded. Git ignore rules apply.")
@staticmethod
def show(title, files):
print(f"{title}:")
for path in sorted(files):
print(f" {path!r}" if not path.isprintable() else f" {path}")
if not files:
print(" (none)")
def status(self, args):
self.git("status", "--untracked-files=no")
watched = self.watches()
if watched:
self.show("Watched files that push will add",
self.files("ls-files", "--others", "--exclude-standard", "-z",
"--", *watched))
def list(self, args):
directory = Path(args.directory).expanduser().resolve()
scope = self.relative(directory)
if not directory.is_dir():
raise ValueError(f"Not a directory: {directory}")
tracked = self.files("ls-files", "--cached", "-z", "--", scope)
untracked_args = ["ls-files", "--others", "--exclude-standard", "-z"]
if self.repo.is_relative_to(self.home):
untracked_args.append(f"--exclude=/{self.repo.relative_to(self.home).as_posix()}/")
untracked = self.files(*untracked_args, "--", scope)
changed = self.files("diff", "--name-only", "--no-renames", "-z", "--", scope)
changed |= self.files("diff", "--cached", "--name-only", "--no-renames", "-z",
"--", scope)
watched = self.watches()
candidates = {path for path in untracked
if any(path.startswith(folder + "/") for folder in watched)}
display = lambda files: {os.path.relpath(self.home / path, directory) for path in files}
self.show("Tracked", display(tracked))
self.show("Untracked (not ignored)", display(untracked - candidates))
self.show("Untracked in watched folders (auto-add on push)", display(candidates))
self.show("Uncommitted changes (push will commit these)", display(changed))
upstream = self.git("rev-parse", "--verify", "@{upstream}", capture=True,
check=False, quiet=True)
if upstream.returncode == 0:
pending = self.files("diff", "--name-only", "--no-renames", "-z",
"@{upstream}...HEAD", "--", scope)
self.show("Committed, not pushed (local upstream reference)", display(pending))
else:
print("Committed, not pushed: unavailable (no upstream configured).")
def push(self, args):
watched = self.watches()
if self.files("ls-files", "--unmerged", "-z"):
raise ValueError("Resolve Git conflicts before you push.")
self.git("add", "--update", "--", ".")
if watched:
new = self.files("ls-files", "--others", "--exclude-standard", "-z", "--", *watched)
if new:
self.git("add", "--pathspec-from-file=-", "--pathspec-file-nul",
input="".join(path + "\0" for path in sorted(new)))
changed = self.git("diff", "--cached", "--quiet", check=False)
if changed.returncode == 1:
self.git("commit", "--allow-empty-message", "-m", args.message)
elif changed.returncode != 0:
raise subprocess.CalledProcessError(changed.returncode, changed.args)
else:
print("No changes to commit.", flush=True)
self.git("push")
def pull(self, args):
dirty = self.git("status", "--porcelain", "--untracked-files=no", capture=True).stdout
stash = None
if dirty:
self.git("stash", "push", "-m", "dotf pull")
stash = self.git("rev-parse", "refs/stash", capture=True).stdout.strip()
try:
pulled = self.git("-c", "rebase.autoStash=false", "-c", "merge.autoStash=false",
"pull", "--ff-only", check=False)
finally:
if stash:
restored = self.git("stash", "apply", "--index", stash, check=False)
if restored.returncode != 0:
print(f"dotf: Restore failed. Your changes remain in stash {stash}.",
file=sys.stderr)
print("Resolve conflicts before you push. Do not apply the stash twice.",
file=sys.stderr)
raise subprocess.CalledProcessError(restored.returncode, restored.args)
latest = self.git("rev-parse", "refs/stash", capture=True, check=False)
if latest.stdout.strip() == stash:
self.git("stash", "drop", "stash@{0}")
else:
print(f"Changes restored. Stash {stash} remains for safety.")
if pulled.returncode != 0:
self.divergence_hint()
raise subprocess.CalledProcessError(pulled.returncode, pulled.args)
def divergence_hint(self):
counts = {}
for key, rev in (("ahead", "@{u}..HEAD"), ("behind", "HEAD..@{u}")):
result = self.git("rev-list", "--count", rev, capture=True, check=False)
if result.returncode != 0:
return
counts[key] = result.stdout.strip()
if counts["behind"] == "0":
return
upstream = self.git("rev-parse", "--abbrev-ref", "@{u}", capture=True, check=False)
print(f"dotf: {counts['ahead']} commit(s) ahead and {counts['behind']} behind "
f"{upstream.stdout.strip()}.", file=sys.stderr)
print("dotf: Replay your commits with 'dotf rebase @{u}'.", file=sys.stderr)
def main():
options = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
options.add_argument("--repo", default="~/.dotfiles.git", help="Git repository (default: ~/.dotfiles.git)")
options.add_argument("--work-tree", default="~", help="Work tree (default: home directory)")
parser = argparse.ArgumentParser(
parents=[options], allow_abbrev=False,
description="Manage dotfiles with a Git repository and your home directory as its work tree.",
epilog="Other commands pass directly to Git. Push auto-adds files recursively in folders with a tracked .dotf marker, except the home directory. Git ignore rules apply.",
)
commands = parser.add_subparsers(dest="command", required=True)
commands.add_parser("pull", help="Stash tracked edits, pull --ff-only, then restore edits and staging")
push = commands.add_parser("push", help="Commit tracked edits and watched files, then push")
push.add_argument("-m", "--message", default="", help="Commit message (default: empty)")
listing = commands.add_parser("list", help="List files and pending changes in the current directory")
listing.add_argument("directory", nargs="?", default=".")
commands.add_parser("status", help="Show Git status and new files in watched folders")
watch = commands.add_parser("watch", help="Show folders with tracked .dotf markers")
watch.add_argument("action", nargs="?", choices=["list"], help="List folders with tracked .dotf markers")
options.add_argument("git_args", nargs=argparse.REMAINDER)
try:
args, unknown = options.parse_known_args()
if not unknown and args.git_args and args.git_args[0] not in commands.choices:
repo = Path(args.repo).expanduser().resolve()
home = Path(args.work_tree).expanduser().resolve()
os.execvp("git", ["git", f"--git-dir={repo}", f"--work-tree={home}", *args.git_args])
args = parser.parse_args()
getattr(Dotfiles(args), args.command)(args)
except (ValueError, OSError) as error:
print(f"dotf: {error}", file=sys.stderr)
return 1
except subprocess.CalledProcessError as error:
return error.returncode
except KeyboardInterrupt:
print("dotf: Interrupted.", file=sys.stderr)
return 130
return 0
if __name__ == "__main__":
sys.exit(main())Conclusion
These scripts might seem random, but they really reflect my everyday workflows, and I can't overstate how much I've come to rely on each and every one of them. If you have cool ones you want to share, or if you have any notes or suggestions, you can reach out to me; I'd love to hear about them.
-
Well, insofar as one can, given that jj currently relies on git as its backend. ↩︎