In the same way, the Japanese 'r' can sound like an 'l', an 'r', or even a 'd' depending on the speaker and non-Japanese listener,
I've always thought it sounds like the shortest possible
Alveolar trill, which is sometimes taught by alternating 'd' and 't' (though the real thing doesn't do that of course), so there's definitely a bit of 'd' in there. Formal romanizations are kind of weird in this respect, since the problem is that only hearing Japanese as a baby, you simply lose the built-in phonemes for 'l' and 'r' - in the same way that probably none of us have the ability to discern all the different click phonemes (used in some African languages).
So you're trying to map something undefined in English to one of several choices, and in some ways it's worse for English to Japanese since both 'l' and 'r' necessarily map to the same Japanese phoneme. At the same time, some Romanizations simply
look better to a native English speaker, presumably by association with other English words - for instance, preferring 'Rei' as a name over 'Lei' even though both are equally valid Romanizations in theory (and I believe 'Lei' is preferred in at least one formal Romanization, which is unfortunate). Earisu/Aeris/Aerith/Earth is another example of this of course, although I'm not sure missing phonemes are the cause of that. Is 'th' hard to discern from 's' for someone who only heard Japanese in their early life or is there just no way to tell them apart in writing?