Okay, I’m being obnoxious with the post title. Granted. And I will concede that there are many good reasons to vote for a particular site over another, one of which is that you think that a given committee will deliver a better convention.
I’m not talking about those reasons.
I know I’m an experienced traveler (and known for same), so I tended to hear people’s travel-related objections to the various proposed Worldcon sites more than other people did.
Here are some of the actual objections I heard about the Finland 2015 Worldcon bid:

I don’t like the TSA

Well, then you should actually only vote for Worldcons outside the US because when you travel there, you’ll only have to deal with the TSA half as much, assuming that your last flight is an international flight. (Example: Helsinki-Frankfurt-San Francisco rather than Helsinki-NYC-San Francisco)
If you don’t have to connect to a domestic flight in the US, then you only have to deal with the TSA on your outbound flight.
Or you could move to San Francisco; we don’t have the TSA there (we have CAS).

I hate the hassles of airport security

So apply for TSA Pre-√. (This assumes Southwest is not your carrier of choice.) For US Citizens and permanent residents, I recommend applying through Global Entry, which also gets you quick immigration. Other programs like NEXUS (Canada) and SENTRI (Mexico) can participate.
And, bonus, Global Entry also means you get the fast immigration line into New Zealand, so you’ll be all set for 2020.
What does Pre-√ get you? The front of the line, even at airports with no Pre-√. The short line (I’ve never seen it more than 4 people long) at airports that do. No taking shoes off. No porno scanner. No unpacking into six bins (I seriously am not exaggerating here, I’ve actually needed six bins more than once). Most people will not need to unpack anything.
On the way back, you can skip the long immigration and customs lines. Stand at the kiosk, answer the questions, look at camera, fingerprint scan, take the receipt, you’re done. It has saved me over 20 minutes at times, though the minimum it’s saved is about a minute and a half.
Recent report from a travel friend, arriving back in the US from Rio:

At IAH (Houston). Sprinted to USCIS (US Customs and Immigration Service) because I’m a noncitizen and I had to beat the São Paulo flight that arrived at the same time as us. Managed to be first in line at immigration, and jetsetr still beat me through using Global Entry after sauntering down from the aircraft.

English isn’t Finland’s first language

English isn’t the first language of aliens, either, but we supposedly love them and crave first contact.
There have been four Worldcons in countries/regions where English was not the first language: Heidelberg (1970), The Hague (1990), Yokohama (2007), and Montreal (2009).
I’d argue that average Finnish command of English easily exceeds that of the average in Montreal. Like the Netherlands, English is very commonly spoken. In fact, I’d argue that the average Finn speaks English at least as well as the average American.

But transit … in a foreign country

Look. I’ve been to a lot of airports in a lot of countries. I think I can safely say that if I can find my way around airports in countries where the non-Roman alphabet makes no sense to me, so can you.
Much as English is the international language of aircraft controllers, almost every sign in almost every airport in the world is in whatever the country’s native language is — and also in English.
Every flight readerboard I’ve ever seen is also in English. Every ATM I’ve seen has English as an option, even in countries that don’t get a lot of American tourists (e.g., Myanmar).

But I’d rather drive/train/bicycle

Fuck Isaac Asimov.
You can have your NASFiC wherever. Let the Worldcon location be freer.
Even Asimov knew how to take a ship. (Hint: Cunard still offers the same transatlantic service it did in Asimov’s days, just less frequent. If you want to go to Europe and don’t want to fly from North America, that (or another line) should be part of your plans.
(For those who don’t know, Asimov never drove or flew. Ever.)

But I need a CPAP on my flight

Get a travel battery. Call the airline, tell them your CPAP’s model number. They will have their medical department clear you. Call to re-confirm 72 hours prior to flight.
It’s not rocket science.
I will admit to having screwed this up once. I’d had a ticket glitch on a United award ticket (during the merger last year) and my clearance got disconnected from the reservation when my ticket blipped out of existence. I’d called to reconfirm one ticket but forgot to check the second. The Swiss airline captain had to call to ground to get clearance. Fortunately, there was documentation on my other non-glitchy reservation. It is possible to get it cleared in flight like that, but I wouldn’t recommend it — it’s awfully embarrassing.
From a perspective of someone who flies a lot — a 10-11 hour flight, like one to Europe, really is the best length. Shorter flights break up sleep habits too much.

I have another medical condition that makes travel difficult

You know what? It happens. Maybe this particular Worldcon isn’t meant to be for you. None of us know for sure we’ll be able to do anything two years hence, so why hold up other people’s fun? Vote “No Preference.”
I know of people who’ve been to Worldcon under some pretty gruesome medical situations — mid-radiation, mid-chemo, and, in the case of a friend, post-terminal diagnosis.
Some conditions are showstoppers for travel, some aren’t. You’d be surprised at what people can travel with, though. I’ve heard stories about extreme medical tourism to Thailand in particular (and if you’ve been to Suvarnabhumi airport and seen the ads, you’ll understand).

Finland’s too expensive

I once heard the Hugos disparagingly described as an award ceremony held by “people who can afford a thousand dollar weekend.” He wasn’t wrong.
San Francisco to Spokane is $546 for next summer. San Francisco to Helsinki’s $1079. Spokane’s room night rate was $139. Helsinki was $80.
For one person, a flight and five nights would therefore wind up being $1251 for Spokane and $1479 for Helsinki. True, Helsinki’s higher, but it’s not as much higher as you might think.
Yeah, but I’m not traveling alone, you say. Fine, for two people sharing a room, Spokane would be $1787 ($894 pp) and Helsinki $2558 ($1279 pp), or $385 more per person.
Or, put another way, from San Francisco, one person going alone to Spokane is pretty much a wash, cost-per-person-wise, with shared accommodation in Helsinki.
Now, I’m not saying the costs aren’t real, or that they’re insignificant. I’m just saying that people were probably not looking at the whole picture or considering that they have two years between now and then.
I’m also going to say: consider the inverse case. Consider how many foreigners would come if it weren’t for the TSA, if costs weren’t so daunting, and if there weren’t language barriers.
I, for one, would like to hear more from the rest of the world, and that means holding Worldcons there.