Index | Thread | Search

From:
Klemens Nanni <kn@openbsd.org>
Subject:
Re: bsd.port.mk: quiet fetch
To:
ports <ports@openbsd.org>
Date:
Fri, 1 Nov 2024 19:53:00 +0000

Download raw body.

Thread
  • Klemens Nanni:

    bsd.port.mk: quiet fetch

  • 20.10.2024 13:58, Klemens Nanni пишет:
    > 06.10.2024 23:52, Klemens Nanni пишет:
    >> 'checksum' is noisy, especially when working on ports with lots of distfiles.
    >> Take net/go-ipfs for example with 1996(!) files in distinfo:
    >>
    >> 	$ cat checksum.sh
    >> 	make checksum "$@" | awk '
    >> 	    /up to date/ {up++; if (up > 1) next}
    >> 	    /SHA256.*OK/ {ok++; if (ok > 1) next}
    >> 	    {print}
    >> 	    END {print "skipped up/ok", up, ok}
    >> 	'
    >>
    >> 	$ sh ./checksum.sh CHECKSUM_QUIET=No
    >> 	===>  Checking files for kubo-0.26.0
    >> 	`/home/distfiles/kubo-v0.26.0.zip' is up to date.
    >> 	>> (SHA256) kubo-v0.26.0.zip: OK
    >> 	skipped up/ok 1996 1996
    >>
    >> In this usual case, I'd like to see all good cases folded so the output fits
    >> on a page, even for monster ports.
    > 
    >> So far so good, I haven't looked into the 'up to date' part, yet.
    > 
    > checksum is quiet now, but fetch is not:
    > 
    > 	$ make fetch | grep -c 'is up to date'
    > 	1996
    > 
    > These lines are produced by make(1) itself;  minimal reproducer (works anywhere,
    > needs no Makefile):
    > 
    > 	$ echo 'int main(){return 0;}' >| foo.c
    > 	$ make foo                             
    > 	cc -O2 -pipe    -o foo foo.c 
    > 	$ make foo
    > 	`foo' is up to date.
    > 
    > Here's a WIP bsd.port.mk diff filtering those lines, such that, finally, even huge
    > ports produce a sane amount of output:
    > 
    > 	$ make clean=all ; make clean=dist
    > 	...
    > 	$ make fetch
    > 	...
    > 	$ make checksum  # runs 'fetch' internally, but now all files are there
    > 	===>  Checking files for kubo-0.26.0
    > 	>> (SHA256) all files: OK
    > 
    > 
    > The "drawback" is that ftp(1) progress meters, if enabled, do not load progressively,
    > since they now run through a pipe and not directly on the TTY anymore.
    > Use 'make _FETCH_FILTER=' with this diff to see old/current behaviour.
    > 
    > One fix would be to teach make(1) how to shut up on such messages.
    > GNU make(1) can do that already:
    > 
    >      -s, --silent, --quiet
    >           Silent operation; do not print the commands as they are executed.
    > 
    > 	$ gmake foo
    > 	gmake: 'foo' is up to date.
    > 	$ gmake foo -s ; echo $?
    > 	0
    > 
    > What do you think about quieting down 'fetch' and 'fetch-all' in general?
    > 
    > Personally, I'd be happy with the simple .mk diff below and not touching make(1),
    > but teaching make(1) a new trick and silencing our targets properly is fun, too.
    
    New version only silencing 'is up to date' under PROGRESS_METER=No, i.e. when there's
    no progressive progress bar in the first place that would turn into a simple line due
    to use of grep/no TTY.
    
    So this includes the diff I just sent in the other thread.
    
    Thoughts?
    
    Index: bsd.port.mk
    ===================================================================
    RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
    diff -u -p -r1.1640 bsd.port.mk
    --- bsd.port.mk	18 Oct 2024 22:39:30 -0000	1.1640
    +++ bsd.port.mk	1 Nov 2024 19:49:08 -0000
    @@ -163,12 +163,17 @@ PKG_DBDIR ?= /var/db/pkg
     
     PROGRESS_METER ?= Yes
     .if ${PROGRESS_METER:L} == "yes"
    -_PROGRESS = -m
    +_PKG_PROGRESS = -m
    +_FETCH_PROGRESS = -m
    +.elif ${PROGRESS_METER:L} == "no"
    +_PKG_PROGRESS =
    +_FETCH_PROGRESS = -M
     .else
    -_PROGRESS =
    +_PKG_PROGRESS =
    +_FETCH_PROGRESS =
     .endif
     
    -FETCH_CMD ?= /usr/bin/ftp -V ${_PROGRESS} -C
    +FETCH_CMD ?= /usr/bin/ftp -V ${_FETCH_PROGRESS} -C
     
     # switch for fetching each distfile: avoid warnings for missing
     # distinfo and wrong size when running makesum
    @@ -181,13 +186,13 @@ PKG_INFO ?= /usr/sbin/pkg_info
     PKG_CREATE ?= /usr/sbin/pkg_create
     PKG_DELETE ?= /usr/sbin/pkg_delete
     
    -_PKG_ADD = ${PKG_ADD} ${_PROGRESS} -I
    -_PKG_CREATE = ${PKG_CREATE} ${_PROGRESS}
    +_PKG_ADD = ${PKG_ADD} ${_PKG_PROGRESS} -I
    +_PKG_CREATE = ${PKG_CREATE} ${_PKG_PROGRESS}
     _SUDO_PKG_ADD_LOCAL = TRUSTED_PKG_PATH=${_PKG_REPO} ${SUDO} ${_PKG_ADD}
     .if ${INSTALL_DEBUG_PACKAGES:L} == "yes"
     _SUDO_PKG_ADD_LOCAL += -d
     .endif
    -_PKG_DELETE = ${PKG_DELETE} ${_PROGRESS}
    +_PKG_DELETE = ${PKG_DELETE} ${_PKG_PROGRESS}
     
     .if defined(PKG_PATH)
     _PKG_PATH = ${PKG_PATH}
    @@ -2525,12 +2530,27 @@ ${_DEP${_m}WANTLIB_COOKIE}: ${_DEPBUILD_
     
     .endfor
     
    +# make(1) has no switch to silence general target status messages,
    +# so the only way to silence them is a pipe/filter, but that means
    +# FETCH_CMD aka. ftp(1) run by recursive MAKE would have no TTY,
    +# i.e. its progress meter wouldn't be progressive.
    +#
    +# gmake(1) -s, --silent, --quiet does skip "... is up to date." messages;
    +# perhaps adopt this behaviour and replace such pipe with just -s?
    +#
    +# Filter only if the user already disabled the only thing we'd "break".
    +.if ${PROGRESS_METER:L} == "no"
    +_FETCH_FILTER =	| { grep -v ' is up to date\.$$' || true; }
    +.else
    +_FETCH_FILTER =
    +.endif
    +
     _internal-fetch-all:
     # See ports/infrastructure/templates/Makefile.template
     	@${ECHO_MSG} "===>  Checking files for ${FULLPKGNAME}${_MASTER}"
     # What FETCH-ALL normally does:
     .  if !empty(MAKESUMFILES)
    -	@${_MAKE} ${MAKESUMFILES:S@^@${DISTDIR}/@}
    +	@${_MAKE} ${MAKESUMFILES:S@^@${DISTDIR}/@} ${_FETCH_FILTER}
     .    endif
     # End of FETCH
     
    @@ -2581,7 +2601,7 @@ _internal-fetch:
     	@${ECHO_MSG} "===>  Checking files for ${FULLPKGNAME}${_MASTER}"
     # What FETCH normally does:
     .  if !empty(CHECKSUMFILES)
    -	@${_MAKE} ${CHECKSUMFILES:S@^@${DISTDIR}/@}
    +	@${_MAKE} ${CHECKSUMFILES:S@^@${DISTDIR}/@} ${_FETCH_FILTER}
     .  endif
     # End of FETCH
     
    
    
    
  • Klemens Nanni:

    bsd.port.mk: quiet fetch