Re: lintpkgsrc 'Out of memory!' error

[matthew sporleder <msporleder%gmail.com@localhost>]

On Mon, Sep 3, 2012 at 7:55 AM, John Marino <netbsd%marino.st@localhost> wrote:
> On 9/3/2012 13:51, matthew sporleder wrote:
>>
>> On Mon, Sep 3, 2012 at 5:36 AM, Marc Espie<espie%nerim.net@localhost>  wrote:
>>>
>>>
>>> Of course, this requires knowing your tool.  Just look through
>>> pkglintsrc,
>>> pass a few suspect data structures thru Devel::Size, and you'll soon
>>> figure
>>> out what's wrong.
>>
>>
>> agreed..
>>
>> how many files do you have in /mech/bulk ?
>
>
> distfile and subdirectories contains 12209 files.

okay..  I -think- this should help.

I didn't profile the program, but i'm pretty sure the listdir() sub
was the cause of your memory issue.
I created 10k distfiles in my dir and it ran on my system, RSS staying under 20M
(there are a lot of efficiencies to be had overall, even in my changes..)


--- lintpkgsrc.pl-orig  2012-09-03 08:08:21.000000000 -0400
+++ lintpkgsrc.pl       2012-09-03 10:50:53.000000000 -0400
@@ -1,4 +1,4 @@
-#! @PERL@
+#! /usr/pkg/bin/perl

 # $NetBSD: lintpkgsrc.pl,v 1.1 2012/07/13 21:12:07 abs Exp $

@@ -601,6 +601,8 @@

 # List (recursive) non directory contents of specified directory
 #
+#TODO this entire sub should be replaced with direct calls to
+#     File::Find
 sub listdir($$) {
     my ( $base, $dir ) = @_;
     my ($thisdir);
@@ -1506,19 +1508,38 @@
         sum  => 'IGNORE'
     };

-    foreach my $file ( listdir( "$pkgdistdir", undef ) ) {
-        my ($dist);
-
-        if ( !defined( $dist = $distfiles{$file} ) ) {
-            $bad_distfiles{$file} = 1;
-
-        }
-        else {
-            if ( $dist->{sum} ne 'IGNORE' ) {
-                push( @{ $sumfiles{ $dist->{sumtype} } }, $file );
+# bad for memory
+#    foreach my $file ( listdir( "$pkgdistdir", undef ) ) {
+#        my ($dist);
+#
+#        if ( !defined( $dist = $distfiles{$file} ) ) {
+#            $bad_distfiles{$file} = 1;
+#
+#        }
+#        else {
+#            if ( $dist->{sum} ne 'IGNORE' ) {
+#                push( @{ $sumfiles{ $dist->{sumtype} } }, $file );
+#            }
+#        }
+#    }
+# check each file in $pkgdistdir
+    find ( { wanted => sub {
+            my ($dist);
+            if ( -f $File::Find::name )
+            {
+                my $distn = $File::Find::name;
+                $distn =~ s/$pkgdistdir\/?//g;
+                if ( !defined ($dist = $distfiles{$distn} ) ) {
+                    $bad_distfiles{$distn} = 1;
+                }
+                else {
+                    if ( $dist->{sum} ne 'IGNORE' ) {
+                        push( @{ $sumfiles{ $dist->{sumtype} } }, $distn );
+                    }
+                }
             }
-        }
-    }
+        } },
+        ($pkgdistdir) );

     if ( $check_unref && %bad_distfiles ) {
         verbose( scalar( keys %bad_distfiles ),