Apparently neither of those tools will shrink the file system, only the partition. The resulting error is something like this when trying to mount:
# btrfs check /dev/nvme0n1p2
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p2
UUID: d7dde9e4-227a-41bb-a96b-830a638d1fc7
[1/7] checking root items
[2/7] checking extents
ERROR: block device size is smaller than total_bytes in device item, has 1505755136 expect >= 1610612736
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 257695744 bytes used, error(s) found
total csum bytes: 250824
total tree bytes: 851968
total fs tree bytes: 360448
total extent tree bytes: 49152
btree space waste bytes: 280984
file data blocks allocated: 256843776
referenced 256843776The solution is to run this on the unmountable device:
btrfs rescue fix-device-size /dev/nvme0n1p2I think this is a bug and I’ve reported it.