AlertDialog broken layout on some devices

I'm experiencing problems with the following approach to create an AlertDialog:

AlertDialog.Builder alertDialog = new AlertDialog.Builder(this)
    .setTitle(R.string.deletion_title)
    .setMessage(R.string.options_deleteExternalConfirm)
    .setPositiveButton(R.string.yes, new DialogInterface.OnClickListener() {
            public void onClick(DialogInterface dialog, int which) {
                Utils.getFolderStructure().mainDir.delete();
                dialog.cancel();
            }
        })
    .setNegativeButton(R.string.no, null);
alertDialog.show();

While the dialog looks ok on most devices, it has layout problems on others (namely the Galaxy S4 - GT-I9505 with Android 5.0.1), see attached picture: . What is the reason for that and how can I fix it?

(Other than adding a padding to a custom dialog when Build.PRODUCT.equals("GT-I9505") since there might be phones out there which have the same issue but which are unknown to me)