[pkgsrc/trunk]: pkgsrc/pkgtools/distlint/files pkgtools/distlint: add initial...

[rillig <rillig%pkgsrc.org@localhost>]

details:   https://anonhg.NetBSD.org/pkgsrc/rev/fc03ef2a2b8d
branches:  trunk
changeset: 383558:fc03ef2a2b8d
user:      rillig <rillig%pkgsrc.org@localhost>
date:      Sat Aug 20 13:32:06 2022 +0000

description:
pkgtools/distlint: add initial draft document, no package yet

diffstat:

 pkgtools/distlint/files/README.md |  122 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 122 insertions(+), 0 deletions(-)

diffs (126 lines):

diff -r 75f14a39b334 -r fc03ef2a2b8d pkgtools/distlint/files/README.md
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/pkgtools/distlint/files/README.md Sat Aug 20 13:32:06 2022 +0000
@@ -0,0 +1,122 @@
+$NetBSD: README.md,v 1.1 2022/08/20 13:32:06 rillig Exp $
+
+# Introduction
+
+Distlint ensures that the distfiles on the TNF servers conform to the
+license requirements.
+
+Distfiles distributed under the GPL must be kept available for as long
+as a binary package based on this distfile is distributed, plus 3
+years.<sup>[citation needed]</sup>
+
+Distfiles from packages with `NO_SRC_ON_FTP` must not be available at all.
+
+Edge case: Imagine a package having `NO_SRC_ON_FTP` and multiple distfiles.
+Some of them must not be available, the others have license GPL.
+
+# Configuration
+
+Distlint is configured by the `distlint.conf` file, which contains one
+or more distdir sections. Each such section configures how a single
+distdir is related to the directories for pkgsrc installations and
+binary package directories:
+
+~~~text
+# Each distdir can be populated by several pkgsrc versions, such as 
+# pkgsrc-current and the quarterly branches.
+# Each distdir can be the source for multiple distributions of binary
+# packages, for example for different platforms. 
+
+distdir /usr/pkgsrc/distfiles
+        database /var/db/distlint/main 
+        pkgsrc /usr/pkgsrc-current
+        pkgsrc /usr/pkgsrc-2022Q2
+        pkgsrc /usr/pkgsrc-2022Q1
+        packages /usr/pkgsrc/packages
+        packages /usr/pkgsrc/current-packages
+
+distdir /pub/pkgsrc-archive/distfiles
+        database /var/db/distlint/archive                
+        pkgsrc /pub/pkgsrc-archive/pkgsrc
+        packages /pub/pkgsrc-archive/packages       
+~~~
+
+# Infrastructure overview
+
+* https://cdn.netbsd.org/pub/pkgsrc/distfiles/
+* https://cdn.netbsd.org/pub/pkgsrc/packages/
+* https://archive.netbsd.org/pub/pkgsrc-archive/distfiles/
+* https://archive.netbsd.org/pub/pkgsrc-archive/packages/
+
+# Approach
+
+Distlint maintains a database of distfile requirements.
+The requirements are collected from all pkgsrc branches that are either
+current or in the archive.
+
+## Examples of database entries
+
+_$distfile_ must not be in distfiles, because on _$updated_at_,
+it belonged to package _$pkgname_ in _$pkgpath_,
+which was marked as `NO_SRC_ON_FTP` because _$no_src_on_ftp_.
+
+_$distfile_ must be kept in distfiles until _$keep_until_,
+because on _$updated_at_, it belonged to package _$pkgname_,
+which is published at _$publish_url_ and licensed under _$license_.
+
+# Implementation details
+
+## NO_SRC_ON_FTP
+
+To find out whether a binary package has `NO_SRC_ON_FTP`, look at its
+`+BUILD_INFO`.
+
+## Find out the distfiles of a binary package
+
+For most binary packages, the file `+BUILD_VERSION` contains the CVS
+revision information of the `distinfo` file.
+
+Some packages use `DISTINFO_FILE` to refer to a `distinfo` file outside
+their PKGPATH. The CVS revision information for these `distinfo` files is
+not recorded anywhere.
+
+Some packages have no `distinfo` file at all because they are self-contained.
+Example: pkgtools/lintpkgsrc.
+
+Whether a binary package had a `distinfo` file or not is not visible from
+looking at the binary package alone.
+
+Using the CVS revision information of the `distinfo` file,
+its file list can be retrieved from CVS.
+
+# Quick hacks
+
+## Find distfiles with NO_SRC_ON_FTP
+
+This program finds most distfiles with NO_SRC_ON_FTP that are referenced
+from the current pkgsrc tree.
+
+Shortcomings:
+
+* It does not find distfiles from stable pkgsrc branches.
+* It does not find distfiles from previous versions of the packages.
+* It does not find distfiles from packages with `DISTINFO_FILE`.
+
+~~~shell
+ssh ftp.netbsd.org
+cd /pub/pkgsrc/current/pkgsrc
+
+for pkgpath in $(grep -r NO_SRC_ON_FTP . 2>/dev/null | cut -d/ -f2-3); do
+
+  if [ -f "$pkgpath/distinfo" ] &&
+    ! grep -r MASTER_SITE_LOCAL "$pkgpath" >/dev/null 2>&1; then
+
+    sed -n 's,^Size (\(.*\)) =.*$,\1,p' "$pkgpath/distinfo" |
+      while read distfile; do
+        if [ -f "/pub/pkgsrc/distfiles/$distfile" ]; then
+          echo "$distfile"
+        fi
+      done
+  fi
+done | sort
+~~~