lib/54955: basesystem's libc++ is too old and contains fixed upstream (crashy) Undefined Behavior

[n54%gmx.com@localhost]

>Number:         54955
>Category:       lib
>Synopsis:       basesystem's libc++ is too old and contains fixed upstream (crashy) Undefined Behavior
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Feb 10 21:55:00 +0000 2020
>Originator:     Kamil Rytarowski
>Release:        NetBSD 9.99.46 amd64
>Organization:
TNF
>Environment:
NetBSD chieftec 9.99.46 NetBSD 9.99.46 (GENERIC) #0: Fri Feb  7 22:09:38 CET 2020  root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC amd64
>Description:
libc++ was updated for the last time almost 5 years ago (!).

commit d03053148da5b5cc0b7cf9f0d2e4441d9a7ac39e
Author: joerg <joerg%NetBSD.org@localhost>
Date:   Thu Aug 20 09:49:32 2015 +0000

    Rerun the import of r245547.

It is also installed into a custom path: `/usr/include/c++/` instead of `/usr/include/c++/v1/`. This modification breaks compatibility with upstream LLVM distribution. New clang's driver for NetBSD supports both `/usr/include/c++/` and `/usr/include/c++/v1/`. It is requested to install lib++ on upgrade to `/usr/include/c++/v1/`.

There are also very good reasons to upgrade libc++:
 - there were compatibility bugs with NetBSD fixed over last years
 - support for recent C++ standard editions
 - stop being close to irrelevantly old for maintaining 3rd party software
>How-To-Repeat:
Try to use __hash_table or __tree. They trigger UBSan.

Presentation of a code that crashes:

"Segfault when assigning map of default-ctr'ed (int, function)"
https://bugs.llvm.org/show_bug.cgi?id=28469

>Fix:
Upgrade libc++ to the modern version.

commit 40492ba417e6cb3da99e8a6aafb80cde0186a96d
Author: Eric Fiselier <eric%efcs.ca@localhost>
Date:   Sat Jul 23 20:36:55 2016 +0000

    Fix undefined behavior in __hash_table
    
    Summary:
    This patch attempts to fix the undefined behavior in __hash_table by changing the node pointer types used throughout. The pointer types are ch
anged for raw pointers in the current ABI and for fancy pointers in ABI V2 (since the fancy pointer types may not be ABI compatible).
    
    The UB in `__hash_table` arises because tree downcasts the embedded end node and then deferences that pointer. Currently there are 2 node type
s in __hash_table:
    
    * `__hash_node_base` which contains the `__next_` pointer.
    * `__hash_node` which contains `__hash_` and `__value_`.
    
    Currently the bucket list, iterators, and `__next_` pointers store pointers to `__hash_node` even though they all need to store `__hash_node_b
ase` pointers.
    This patch makes that change by introducing a `__next_pointer` typedef which is a pointer to `__hash_node` in the current ABI and `__hash_node
_base` afterwards.
    
    One notable change is to the type of `__bucket_list` which used to be defined as `unique_ptr<__node_pointer[], ...>` and is now `unique_ptr<__
next_pointer[], ...>` meaning that we now allocate and deallocate different types using a different allocator. I'm going to give this part of the 
change more thought since it may introduce compatibility issues.
    
    This change is similar to D20786.
    
    
    
    Reviewers: mclow.lists, EricWF
    
    Subscribers: cfe-commits
    
    Differential Revision: https://reviews.llvm.org/D20787
    
    llvm-svn: 276533

commit d05b10ab4fc6514caaddd3f08e798dcd9ac0ddfc
Author: Eric Fiselier <eric%efcs.ca@localhost>
Date:   Tue Jul 19 17:56:20 2016 +0000

    Fix undefined behavior in __tree
    
    Summary:
    This patch attempts to fix the undefined behavior in __tree by changing the node pointer types used throughout. The pointer types are changed for raw pointers in the current ABI and for fancy pointers in ABI V2 (since the fancy pointer types may not be ABI compatible).
    
    The UB in `__tree` arises because tree downcasts the embedded end node and then deferences that pointer. Currently there are 3 node types in __tree.
    
    * `__tree_end_node` which contains the `__left_` pointer. This node is embedded within the container.
    * `__tree_node_base` which contains `__right_`, `__parent_` and `__is_black`. This node is used throughout the tree rebalancing algorithms.
    * `__tree_node` which contains `__value_`.
    
    Currently `__tree` stores the start of the tree, `__begin_node_`, as a pointer to a `__tree_node`. Additionally the iterators store their position as a pointer to a `__tree_node`. In both of these cases the pointee can be the end node. This is fixed by changing them to store `__tree_end_node` pointers instead.
    
    To make this change I introduced an `__iter_pointer` typedef which is defined to be a pointer to either `__tree_end_node` in the new ABI or `__tree_node` in the current one.
    Both `__tree::__begin_node_` and iterator pointers are now stored as `__iter_pointers`.
    
    The other situation where `__tree_end_node` is stored as the wrong type is in `__tree_node_base::__parent_`.  Currently `__left_`, `__right_`, and `__parent_` are all `__tree_node_base` pointers. Since the end node will only be stored in `__parent_` the fix is to change `__parent_` to be a pointer to `__tree_end_node`.
    
    To make this change I introduced a `__parent_pointer` typedef which is defined to be a pointer to either `__tree_end_node` in the new ABI or `__tree_node_base` in the current one.
    
    Note that in the new ABI `__iter_pointer` and `__parent_pointer` are the same type (but not in the old one). The confusion between these two types is unfortunate but it was the best solution I could come up with that maintains the ABI.
    
    The typedef changes force a ton of explicit type casts to correct pointer types and to make current code compatible with both the old and new pointer typedefs. This is the bulk of the change and it's really messy. Unfortunately I don't know how to avoid it.
    
    Please let me know what you think.
    
    
    
    
    
    Reviewers: howard.hinnant, mclow.lists
    
    Subscribers: howard.hinnant, bbannier, cfe-commits
    
    Differential Revision: https://reviews.llvm.org/D20786
    
    llvm-svn: 276003