[src/netbsd-9]: src/common/lib/libc/arch/aarch64/atomic Pull up following rev...

To: source-changes-hg%NetBSD.org@localhost
Subject: [src/netbsd-9]: src/common/lib/libc/arch/aarch64/atomic Pull up following rev...
From: martin <martin%NetBSD.org@localhost>
Date: Wed, 11 Aug 2021 21:28:32 +0000
details:   https://anonhg.NetBSD.org/src/rev/9da215fd7538
branches:  netbsd-9
changeset: 985187:9da215fd7538
user:      martin <martin%NetBSD.org@localhost>
date:      Wed Aug 11 17:19:01 2021 +0000

description:
Pull up following revision(s) (requested by skrll in ticket #1331):

        common/lib/libc/arch/aarch64/atomic/atomic_add_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_or_64.S: revision 1.3
        common/lib/libc/arch/aarch64/atomic/atomic_and_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_and_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_add_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h: revision 1.6
        common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.4
        common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S: revision 1.6
        common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_or_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_or_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_or_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S: revision 1.4
        common/lib/libc/arch/aarch64/atomic/atomic_add_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S: revision 1.4
        common/lib/libc/arch/aarch64/atomic/atomic_add_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S: revision 1.4
        common/lib/libc/arch/aarch64/atomic/membar_ops.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S: revision 1.5
        common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_and_8.S: revision 1.2
        common/lib/libc/arch/aarch64/atomic/atomic_and_64.S: revision 1.2

Part I of ad@'s performance improvements for aarch64
- Remove memory barriers from the atomic ops.  I don't understand why thos=
e
   are there.  Is it some architectural thing, or for a CPU bug, or just
   over-caution maybe?  They're not needed for correctness.
- Have unlikely conditional branches go forwards to help the static branch
   predictor.

Remove memory barriers from the atomic ops macros in the same way as was
done for the other atomic ops earlier.

Use the correct barriers - all of membar_{sync,producer,consumer} have
less scope than before.

LGTM from riastradh

As we're providing the legacy gcc __sync built-in functions for atomic
memory access we might as well get the memory barriers right...

 From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)
    This built-in function is not a full barrier, but rather an acquire
    barrier. This means that references after the operation cannot move to
    (or be speculated to) before the operation, but previous memory stores
    may not be globally visible yet, and previous memory loads may not yet
    be satisfied.

void __sync_lock_release (type *ptr, ...)
    This built-in function is not a full barrier, but rather a release
    barrier. This means that all previous memory stores are globally
    visible, and all previous memory loads have been satisfied, but
    following memory reads are not prevented from being speculated to
    before the barrier.

diffstat:

 common/lib/libc/arch/aarch64/atomic/atomic_add_16.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_add_32.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_add_64.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_add_8.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_and_16.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_and_32.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_and_64.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_and_8.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S  |   19 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S  |   20 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_cas_64.S  |   20 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_cas_8.S   |   20 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_dec_32.S  |   10 +-
 common/lib/libc/arch/aarch64/atomic/atomic_dec_64.S  |   10 +-
 common/lib/libc/arch/aarch64/atomic/atomic_inc_32.S  |   10 +-
 common/lib/libc/arch/aarch64/atomic/atomic_inc_64.S  |   10 +-
 common/lib/libc/arch/aarch64/atomic/atomic_nand_16.S |   40 +++++-
 common/lib/libc/arch/aarch64/atomic/atomic_nand_32.S |   48 ++++++--
 common/lib/libc/arch/aarch64/atomic/atomic_nand_64.S |   40 +++++-
 common/lib/libc/arch/aarch64/atomic/atomic_nand_8.S  |   41 +++++-
 common/lib/libc/arch/aarch64/atomic/atomic_op_asm.h  |  106 +++++++++++++++++-
 common/lib/libc/arch/aarch64/atomic/atomic_or_16.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_or_32.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_or_64.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_or_8.S    |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_sub_16.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_sub_32.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_sub_64.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_sub_8.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_swap_16.S |   16 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_swap_32.S |   16 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_swap_64.S |   16 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_swap_8.S  |   16 ++-
 common/lib/libc/arch/aarch64/atomic/atomic_xor_16.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_xor_32.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_xor_64.S  |    8 +-
 common/lib/libc/arch/aarch64/atomic/atomic_xor_8.S   |    8 +-
 common/lib/libc/arch/aarch64/atomic/membar_ops.S     |   18 +-
 38 files changed, 473 insertions(+), 163 deletions(-)

diffs (truncated from 1558 to 300 lines):

diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_add_16.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_add_16.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_add_16.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_add_16.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_add_16.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP16(add, add)
 
+SYNC_FETCH_OP16(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_16,_atomic_add_16)
 ATOMIC_OP_ALIAS(atomic_add_short,_atomic_add_16)
-STRONG_ALIAS(__sync_fetch_and_add_2,_atomic_add_16)
 STRONG_ALIAS(_atomic_add_short,_atomic_add_16)
 
 ATOMIC_OP16_NV(add, add)
 
+SYNC_OP16_FETCH(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_16_nv,_atomic_add_16_nv)
 ATOMIC_OP_ALIAS(atomic_add_short_nv,_atomic_add_16_nv)
-STRONG_ALIAS(__sync_add_and_fetch_2,_atomic_add_16_nv)
 STRONG_ALIAS(_atomic_add_short_nv,_atomic_add_16_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_add_32.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_add_32.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_add_32.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_add_32.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_add_32.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP32(add, add)
 
+SYNC_FETCH_OP32(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_32,_atomic_add_32)
 ATOMIC_OP_ALIAS(atomic_add_int,_atomic_add_32)
-STRONG_ALIAS(__sync_fetch_and_add_4,_atomic_add_32)
 STRONG_ALIAS(_atomic_add_int,_atomic_add_32)
 
 ATOMIC_OP32_NV(add, add)
 
+SYNC_OP32_FETCH(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_32_nv,_atomic_add_32_nv)
 ATOMIC_OP_ALIAS(atomic_add_int_nv,_atomic_add_32_nv)
-STRONG_ALIAS(__sync_add_and_fetch_4,_atomic_add_32_nv)
 STRONG_ALIAS(_atomic_add_int_nv,_atomic_add_32_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_add_64.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_add_64.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_add_64.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_add_64.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_add_64.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,18 +33,20 @@
 
 ATOMIC_OP64(add, add)
 
+SYNC_FETCH_OP64(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_64_nv,_atomic_add_64_nv)
 ATOMIC_OP_ALIAS(atomic_add_ptr_nv,_atomic_add_64_nv)
 ATOMIC_OP_ALIAS(atomic_add_long_nv,_atomic_add_64_nv)
 STRONG_ALIAS(_atomic_add_ptr_nv,_atomic_add_64_nv)
 STRONG_ALIAS(_atomic_add_long_nv,_atomic_add_64_nv)
-STRONG_ALIAS(__sync_add_and_fetch_8,_atomic_add_64_nv)
 
 ATOMIC_OP64_NV(add, add)
 
+SYNC_OP64_FETCH(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_64,_atomic_add_64)
 ATOMIC_OP_ALIAS(atomic_add_ptr,_atomic_add_64)
 ATOMIC_OP_ALIAS(atomic_add_long,_atomic_add_64)
 STRONG_ALIAS(_atomic_add_ptr,_atomic_add_64)
 STRONG_ALIAS(_atomic_add_long,_atomic_add_64)
-STRONG_ALIAS(__sync_fetch_and_add_8,_atomic_add_64)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_add_8.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_add_8.S        Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_add_8.S        Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_add_8.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_add_8.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP8(add, add)
 
+SYNC_FETCH_OP8(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_8,_atomic_add_8)
 ATOMIC_OP_ALIAS(atomic_add_char,_atomic_add_8)
-STRONG_ALIAS(__sync_fetch_and_add_1,_atomic_add_8)
 STRONG_ALIAS(_atomic_add_char,_atomic_add_8)
 
 ATOMIC_OP8_NV(add, add)
 
+SYNC_OP8_FETCH(add, add)
+
 ATOMIC_OP_ALIAS(atomic_add_8_nv,_atomic_add_8_nv)
 ATOMIC_OP_ALIAS(atomic_add_char_nv,_atomic_add_8_nv)
-STRONG_ALIAS(__sync_add_and_fetch_1,_atomic_add_8_nv)
 STRONG_ALIAS(_atomic_add_char_nv,_atomic_add_8_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_and_16.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_and_16.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_and_16.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_and_16.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_and_16.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP16(and, and)
 
+SYNC_FETCH_OP16(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_16,_atomic_and_16)
 ATOMIC_OP_ALIAS(atomic_and_ushort,_atomic_and_16)
-STRONG_ALIAS(__sync_fetch_and_and_2,_atomic_and_16)
 STRONG_ALIAS(_atomic_and_ushort,_atomic_and_16)
 
 ATOMIC_OP16_NV(and, and)
 
+SYNC_OP16_FETCH(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_16_nv,_atomic_and_16_nv)
 ATOMIC_OP_ALIAS(atomic_and_ushort_nv,_atomic_and_16_nv)
-STRONG_ALIAS(__sync_and_and_fetch_2,_atomic_and_16_nv)
 STRONG_ALIAS(_atomic_and_ushort_nv,_atomic_and_16_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_and_32.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_and_32.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_and_32.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_and_32.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_and_32.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP32(and, and)
 
+SYNC_FETCH_OP32(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_32,_atomic_and_32)
 ATOMIC_OP_ALIAS(atomic_and_uint,_atomic_and_32)
-STRONG_ALIAS(__sync_fetch_and_and_4,_atomic_and_32)
 STRONG_ALIAS(_atomic_and_uint,_atomic_and_32)
 
 ATOMIC_OP32_NV(and, and)
 
+SYNC_OP32_FETCH(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_32_nv,_atomic_and_32_nv)
 ATOMIC_OP_ALIAS(atomic_and_uint_nv,_atomic_and_32_nv)
-STRONG_ALIAS(__sync_and_and_fetch_4,_atomic_and_32_nv)
 STRONG_ALIAS(_atomic_and_uint_nv,_atomic_and_32_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_and_64.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_and_64.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_and_64.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_and_64.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_and_64.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP64(and, and)
 
+SYNC_FETCH_OP64(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_64,_atomic_and_64)
 ATOMIC_OP_ALIAS(atomic_and_ulong,_atomic_and_64)
 STRONG_ALIAS(_atomic_and_ulong,_atomic_and_64)
-STRONG_ALIAS(__sync_fetch_and_and_8,_atomic_and_64)
 
 ATOMIC_OP64_NV(and, and)
 
+SYNC_OP64_FETCH(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_64_nv,_atomic_and_64_nv)
 ATOMIC_OP_ALIAS(atomic_and_ulong_nv,_atomic_and_64_nv)
 STRONG_ALIAS(_atomic_and_ulong_nv,_atomic_and_64_nv)
-STRONG_ALIAS(__sync_and_and_fetch_8,_atomic_and_64_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_and_8.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_and_8.S        Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_and_8.S        Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_and_8.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_and_8.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -33,14 +33,16 @@
 
 ATOMIC_OP8(and, and)
 
+SYNC_FETCH_OP8(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_8,_atomic_and_8)
 ATOMIC_OP_ALIAS(atomic_and_uchar,_atomic_and_8)
-STRONG_ALIAS(__sync_fetch_and_and_1,_atomic_and_8)
 STRONG_ALIAS(_atomic_and_uchar,_atomic_and_8)
 
 ATOMIC_OP8_NV(and, and)
 
+SYNC_OP8_FETCH(and, and)
+
 ATOMIC_OP_ALIAS(atomic_and_8_nv,_atomic_and_8_nv)
 ATOMIC_OP_ALIAS(atomic_and_uchar_nv,_atomic_and_8_nv)
-STRONG_ALIAS(__sync_and_and_fetch_1,_atomic_and_8_nv)
 STRONG_ALIAS(_atomic_and_uchar_nv,_atomic_and_8_nv)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_cas_16.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_cas_16.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_cas_16.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -37,12 +37,25 @@
        cmp     w0, w1                  /*   compare? */
        b.ne    2f
        stxrh   w3, w2, [x4]            /* store new value */
-       cbnz    w3, 1b                  /*   succeed? nope, try again. */
-       dmb     st                      /* data memory barrier */
+       cbnz    w3, 3f                  /*   succeed? nope, try again. */
 2:     ret                             /* return. */
+3:     b       1b
 END(_atomic_cas_16)
 
 ATOMIC_OP_ALIAS(atomic_cas_16,_atomic_cas_16)
 STRONG_ALIAS(_atomic_cas_short,_atomic_cas_16)
 STRONG_ALIAS(_atomic_cas_ushort,_atomic_cas_16)
 STRONG_ALIAS(__sync_val_compare_and_swap_2,_atomic_cas_16)
+
+ENTRY_NP(__sync_val_compare_and_swap_2)
+       mov     x4, x0                  /* we need x0 for return value */
+       dmb     ish
+1:     ldxrh   w0, [x4]                /* load old value */
+       cmp     w0, w1                  /*   compare? */
+       b.ne    2f
+       stxrh   w3, w2, [x4]            /* store new value */
+       cbnz    w3, 3f                  /*   succeed? nope, try again. */
+       dmb     ish
+2:     ret                             /* return. */
+3:     b       1b
+END(__sync_val_compare_and_swap_2)
diff -r 5f8fe4e56137 -r 9da215fd7538 common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S
--- a/common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S       Wed Aug 11 17:05:42 2021 +0000
+++ b/common/lib/libc/arch/aarch64/atomic/atomic_cas_32.S       Wed Aug 11 17:19:01 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic_cas_32.S,v 1.1 2014/08/10 05:47:35 matt Exp $ */
+/* $NetBSD: atomic_cas_32.S,v 1.1.28.1 2021/08/11 17:19:01 martin Exp $ */
 
 /*-
  * Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -37,16 +37,28 @@
        cmp     w0, w1                  /*   compare? */
        b.ne    2f                      /*     return if different */
        stxr    w3, w2, [x4]            /* store new value */
-       cbnz    w3, 1b                  /*   succeed? nope, try again. */
-       dmb     st
+       cbnz    w3, 3f                  /*   succeed? nope, try again. */
 2:     ret                             /* return. */
+3:     b       1b
 END(_atomic_cas_32)
 
 ATOMIC_OP_ALIAS(atomic_cas_32,_atomic_cas_32)
 ATOMIC_OP_ALIAS(atomic_cas_uint,_atomic_cas_32)
 ATOMIC_OP_ALIAS(atomic_cas_32_ni,_atomic_cas_32)
 ATOMIC_OP_ALIAS(atomic_cas_uint_ni,_atomic_cas_32)
-STRONG_ALIAS(__sync_val_compare_and_swap_4,_atomic_cas_32)
 STRONG_ALIAS(_atomic_cas_uint,_atomic_cas_32)
 STRONG_ALIAS(_atomic_cas_32_ni,_atomic_cas_32)
 STRONG_ALIAS(_atomic_cas_uint_ni,_atomic_cas_32)
+
+ENTRY_NP(__sync_val_compare_and_swap_4)
+       mov     x4, x0                  /* we need x0 for return value */
+       dmb     ish
+1:     ldxr    w0, [x4]                /* load old value */
+       cmp     w0, w1                  /*   compare? */
+       b.ne    2f                      /*     return if different */
Prev by Date: [src/netbsd-9]: src/sys/netinet6 Pull up following revision(s) (requested by ...
Next by Date: [src/netbsd-9]: src/doc Tickets #1330 - #1332
Previous by Thread: [src/netbsd-9]: src/sys/netinet6 Pull up following revision(s) (requested by ...
Next by Thread: [src/netbsd-9]: src/doc Tickets #1330 - #1332
Indexes:
Home | Main Index | Thread Index | Old Index