From: Stuart Henderson <stu@spacehopper.org>
Subject: Re: distfiles/by_cipher dirs
To: ports <ports@openbsd.org>
Date: Thu, 1 Oct 2026 12:04:08 +0100

(obviously post unlock)

On 2026/09/30 19:23, Marc Espie wrote:
> On Wed, Sep 30, 2026 at 01:44:41PM +0100, Stuart Henderson wrote:
> > 
> > ...however, by_cipher is only used by ports infrastructure when fetching
> > files from ftp.openbsd.org etc, or when dpb fetches files. Specifically
> > when _building_, ports only looks in ${FULLDISTDIR}/file, not
> > ${DISTDIR}/by_ciphers.

I was wrong here, it's not normally used for fetching from ftp.o.o as
a backup SITES - I think it's actually only used for REFETCH=true,
which is currently broken. It tries to fetch from a url like
https://<...>/pub/OpenBSD/distfiles/by_cipher/sha256/xx/xxx<...>xxx=/
i.e. with a trailing / and no filename, so AFAICT the files in by_cipher
on the mirrors are not used at all by ports infra.

But now I've seen what REFETCH is for, it doesn't seem a bad idea to
make it work again, it could simplify diffing old and new files when
upstream re-rolls something, which probably happens more these days than
when it was first invented.

Here's an example of how it could look, just touching link-checksums
and bsd.port.mk so far. Since current files are not used anyway I went
with my idea of .../sha256/xx/xxxxxx as a file rather than a dir which
should be a lot nicer to servers, and s,/,_,g.

Obviously dpb and clean-old-distfiles needed too if this is a sensible
path to take...

Index: fetch/link-checksums
===================================================================
RCS file: /cvs/ports/infrastructure/fetch/link-checksums,v
diff -u -p -r1.4 link-checksums
--- fetch/link-checksums	29 Jul 2008 18:00:19 -0000	1.4
+++ fetch/link-checksums	1 Oct 2026 10:51:16 -0000
@@ -10,12 +10,12 @@

 DISTDIR=${DISTDIR:-/usr/ports/distfiles}
 cd ${DISTDIR}
-CIPHERS=${CIPHERS:-sha1 md5 rmd160 sha256}
+HASHES=${HASHES:-sha256}

-bdir=by_cipher
-# Build the find so that existing cipher dirs are avoided
+bdir=by_hash
+# Build the find so that existing hash dirs are avoided
 exclude="-type d -name $bdir -prune"
-for ci in ${CIPHERS}
+for ci in ${HASHES}
 do
 	exclude="${exclude} -o -type d -name $ci -prune"
 done
@@ -23,12 +23,12 @@ echo "find . \( $exclude \) -o -type f -
 find . \( $exclude \) -o -type f -print | while read i
 do
 	file=`basename $i`
-	for ci in ${CIPHERS}
+	for ci in ${HASHES}
 	do
-		result=`cksum -a $ci -b <$i`
-		res=`echo $result|sed -e 's,\(..\).*,\1,'`
-		dir=$bdir/$ci/$res/$result
+		result=`cksum -a $ci -b <$i | sed -e 's,/,_,g'`
+		res=`echo $result | sed -e 's,\(..\).*,\1,'`
+		dir=$bdir/$ci/$res
 		mkdir -p $dir
-		ln -f $i $dir/$file
+		ln -f $i $dir/$result
 	done
 done
Index: mk/bsd.port.mk
===================================================================
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
diff -u -p -r1.1652 bsd.port.mk
--- mk/bsd.port.mk	19 Sep 2026 23:17:29 -0000	1.1652
+++ mk/bsd.port.mk	1 Oct 2026 10:51:16 -0000
@@ -3293,13 +3293,13 @@ _internal-subpackage: ${_PACKAGE_COOKIES
 _REFETCH_INFO ?=

 # list is pretty much self explanatory
-.for f cipher value in ${_REFETCH_INFO}
+.for f hash value in ${_REFETCH_INFO}
 ${DISTDIR}/$f:
 	@lock=${@:T}.dist; ${_SIMPLE_LOCK}; \
 	${_PFETCH} install -d ${DISTDIR_MODE} ${@:H}; \
 	cd ${@:H}; \
 	file=$@.part; \
-	for site in ${SITE_OPENBSD:=by_cipher/${cipher}/${value:C/(..).*/\1/}/${value}/} ${SITE_OPENBSD:=${cipher}/${value}/}; do \
+	for site in ${SITE_OPENBSD:=by_hash/${hash}/${value:S,/,_,g:C/(..).*/\1/}/${value:S,/,_,g}}; do \
 		${ECHO_MSG} ">> Fetch $${site}$u"; \
 		if ${_PFETCH} ${FETCH_CMD} -o $$file $${site}$u; then \
 			ck=`${_size_fragment} $$file $p`; \