CVS commit: pkgsrc/devel/jemalloc

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Wed Apr 24 17:11:00 UTC 2019

Modified Files:
        pkgsrc/devel/jemalloc: Makefile distinfo
Added Files:
        pkgsrc/devel/jemalloc/patches:
            patch-include_jemalloc_internal_quantum.h
Removed Files:
        pkgsrc/devel/jemalloc/patches:
            patch-include_jemalloc_internal_jemalloc_internal_types.h

Log Message:
jemalloc: updated to 5.2.0

5.2.0:
This release includes a few notable improvements, which are summarized below: 1) improved fast-path performance from the optimizations by @djwatson; 2) reduced virtual memory fragmentation and 
metadata usage; and 3) bug fixes on setting the number of background threads. In addition, peak / spike memory usage is improved with certain allocation patterns. As usual, the release and prior dev 
versions have gone through large-scale production testing.

New features:
Implement oversize_threshold, which uses a dedicated arena for allocations crossing the specified threshold to reduce fragmentation.
Add extents usage information to stats.
Log time information for sampled allocations.
Support 0 size in sdallocx.
Output rate for certain counters in malloc_stats.
Add configure option --enable-readlinkat, which allows the use of readlinkat over readlink.
Add configure options --{enable,disable}-{static,shared} to allow not building unwanted libraries.
Add configure option --disable-libdl to enable fully static builds.
Add mallctl interfaces:
opt.oversize_threshold
stats.arenas.<i>.extent_avail
stats.arenas.<i>.extents.<j>.n{dirty,muzzy,retained}
stats.arenas.<i>.extents.<j>.{dirty,muzzy,retained}_bytes

Portability improvements:
Update MSVC builds.
Workaround a compiler optimizer bug on s390x.
Make use of pthread_set_name_np(3) on FreeBSD.
Implement malloc_getcpu() to enable percpu_arena for windows.
Link against -pthread instead of -lpthread.
Make background_thread not dependent on libdl.
Add stringify to fix a linker directive issue on MSVC.
Detect and fall back when 8-bit atomics are unavailable.
Fall back to the default pthread_create(3) if dlsym(3) fails.

Optimizations and refactors:
Refactor the TSD module.
Avoid taking extents_muzzy mutex when muzzy is disabled.
Avoid taking large_mtx for auto arenas on the tcache flush path.
Optimize ixalloc by avoiding a size lookup.
Implement opt.oversize_threshold which uses a dedicated arena for requests crossing the threshold, also eagerly purges the oversize extents. Default the threshold to 8 MiB.
Clean compilation with -Wextra.
Refactor the size class module.
Refactor the stats emitter.
Optimize pow2_ceil.
Avoid runtime detection of lazy purging on FreeBSD.
Optimize mmap(2) alignment handling on FreeBSD.
Improve error handling for THP state initialization.
Rework the malloc() fast path.
Rework the free() fast path.
Refactor and optimize the tcache fill / flush paths.
Optimize sync / lwsync on PowerPC.
Bypass extent_dalloc() when retain is enabled.
Optimize the locking on large deallocation.
Reduce the number of pages committed from sanity checking in debug build.
Deprecate OSSpinLock.
Lower the default number of background threads to 4 (when the feature is enabled).
Optimize the trylock spin wait.
Use arena index for arena-matching checks.
Avoid forced decay on thread termination when using background threads.
Disable muzzy decay by default.
Only initialize libgcc unwinder when profiling is enabled.

Bug fixes (all only relevant to jemalloc 5.x):
Fix background thread index issues with max_background_threads.
Fix stats output for opt.lg_extent_max_active_fit.
Fix opt.prof_prefix initialization.
Properly trigger decay on tcache destroy.
Fix tcache.flush.
Detect whether explicit extent zero out is necessary with huge pages or custom extent hooks, which may change the purge semantics.
Fix a side effect caused by extent_max_active_fit combined with decay-based purging, where freed extents can accumulate and not be reused for an extended period of time.
Fix a missing unlock on extent register error handling.

Testing:
Simplify the Travis script output.
Update the test scripts for FreeBSD.
Add unit tests for the producer-consumer pattern.
Add Cirrus-CI config for FreeBSD builds.
Add size-matching sanity checks on tcache flush.

Incompatible changes:
Remove --with-lg-page-sizes.

Documentation:
Attempt to build docs by default, however skip doc building when xsltproc is missing.


To generate a diff of this commit:
cvs rdiff -u -r1.13 -r1.14 pkgsrc/devel/jemalloc/Makefile
cvs rdiff -u -r1.12 -r1.13 pkgsrc/devel/jemalloc/distinfo
cvs rdiff -u -r1.1 -r0 \
    pkgsrc/devel/jemalloc/patches/patch-include_jemalloc_internal_jemalloc_internal_types.h
cvs rdiff -u -r0 -r1.1 \
    pkgsrc/devel/jemalloc/patches/patch-include_jemalloc_internal_quantum.h

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/devel/jemalloc/Makefile
diff -u pkgsrc/devel/jemalloc/Makefile:1.13 pkgsrc/devel/jemalloc/Makefile:1.14
--- pkgsrc/devel/jemalloc/Makefile:1.13 Sun Mar 17 13:28:22 2019
+++ pkgsrc/devel/jemalloc/Makefile      Wed Apr 24 17:10:59 2019
@@ -1,7 +1,6 @@
-# $NetBSD: Makefile,v 1.13 2019/03/17 13:28:22 rin Exp $
+# $NetBSD: Makefile,v 1.14 2019/04/24 17:10:59 adam Exp $
 
-DISTNAME=      jemalloc-5.1.0
-PKGREVISION=   2
+DISTNAME=      jemalloc-5.2.0
 CATEGORIES=    devel
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=jemalloc/}
 GITHUB_RELEASE=        ${PKGVERSION_NOREV}

Index: pkgsrc/devel/jemalloc/distinfo
diff -u pkgsrc/devel/jemalloc/distinfo:1.12 pkgsrc/devel/jemalloc/distinfo:1.13
--- pkgsrc/devel/jemalloc/distinfo:1.12 Fri Mar 22 16:30:02 2019
+++ pkgsrc/devel/jemalloc/distinfo      Wed Apr 24 17:10:59 2019
@@ -1,10 +1,10 @@
-$NetBSD: distinfo,v 1.12 2019/03/22 16:30:02 rin Exp $
+$NetBSD: distinfo,v 1.13 2019/04/24 17:10:59 adam Exp $
 
-SHA1 (jemalloc-5.1.0.tar.bz2) = 92fdc0b38680aaee1fa7ccd89cbf1af61224ff46
-RMD160 (jemalloc-5.1.0.tar.bz2) = 52b1340ca8cafd7414aa5e3a7a028cb1feba0b80
-SHA512 (jemalloc-5.1.0.tar.bz2) = d9abebe54d303ca931b8c31c1033f23ff5fb060f2377ec8386f4d79c352e65c78ed34f680c352dac14f7d7115d10245782d553d988bc13df2eb34a2f0942ef6f
-Size (jemalloc-5.1.0.tar.bz2) = 515622 bytes
+SHA1 (jemalloc-5.2.0.tar.bz2) = 5a5255f7bca3e79a063f26f292cf93f17fe3b14f
+RMD160 (jemalloc-5.2.0.tar.bz2) = a805e8c2ad3e7a2f7960bc1407baeaca797bd77e
+SHA512 (jemalloc-5.2.0.tar.bz2) = e3be4d534770126caf10f2684aed9fe4ba1422dd47625fe50343cfb750f26eff869fcc7d1e30a96dd6c73f6614c4bbcd560fd24fc26b55ac731c43e60fd05234
+Size (jemalloc-5.2.0.tar.bz2) = 543892 bytes
 SHA1 (patch-Makefile.in) = 62e04375dec9aab678fed5bc32559deb942e799f
 SHA1 (patch-configure) = 5dec3fb5b2ece549e40743780db3057dd83cc17e
-SHA1 (patch-include_jemalloc_internal_jemalloc_internal_types.h) = 07f2d661d8141152fc7c80f24f9dab4be90cb4b3
+SHA1 (patch-include_jemalloc_internal_quantum.h) = e291b59958813fc1af28bea0a5a63d5715090a48
 SHA1 (patch-src_pages.c) = 57d5f743efc8d9a5c5e2619b0acde08b3404659c

Added files:

Index: pkgsrc/devel/jemalloc/patches/patch-include_jemalloc_internal_quantum.h
diff -u /dev/null pkgsrc/devel/jemalloc/patches/patch-include_jemalloc_internal_quantum.h:1.1
--- /dev/null   Wed Apr 24 17:11:00 2019
+++ pkgsrc/devel/jemalloc/patches/patch-include_jemalloc_internal_quantum.h     Wed Apr 24 17:11:00 2019
@@ -0,0 +1,15 @@
+$NetBSD: patch-include_jemalloc_internal_quantum.h,v 1.1 2019/04/24 17:11:00 adam Exp $
+
+Add 32-bit SPARC support.
+
+--- include/jemalloc/internal/quantum.h.orig   2019-04-24 16:37:33.000000000 +0000
++++ include/jemalloc/internal/quantum.h
+@@ -15,7 +15,7 @@
+ #  ifdef __alpha__
+ #    define LG_QUANTUM                4
+ #  endif
+-#  if (defined(__sparc64__) || defined(__sparcv9) || defined(__sparc_v9__))
++#  if (defined(__sparc64__) || defined(__sparcv9) || defined(__sparc_v9__) || defined(__sparc__))
+ #    define LG_QUANTUM                4
+ #  endif
+ #  if (defined(__amd64__) || defined(__x86_64__) || defined(_M_X64))