Mostly revert ebbc3140ca0d7eee154f7a67ccdae7d3d88d13fd.
We don't need to special-case anything for arm64, the check for the pointer
size is already done for us, just keep the bits about having arm and arm64
having to add padding for 32bits binaries.
MFC after: 1 week
f4b7018af1