Tips for a better first-time hackathon experience

Here are a few things you can consider. These are by no means the only way to go about hackathons, just what's worked for me.

Project
I strongly strongly recommend doing something you haven't done before. Even if you're going into a hackathon with the intention of winning, trying something new is always fun and will keep you engaged throughout the event. Jump into mobile development, try hardware hacking, write a compiler, whatever; it's always fun to try out new things.
Sticking to things you're comfortable with IMO limits your creativity.

Design
Strike a balance. This is a hackathon, judges aren't expecting the most polished and clean UI, but it helps if your app is very UI heavy. I've seem some amazing front end hacks that absolutely wow the judges. What I want to discourage is spending an hour on a logo or wasting precious time because your columns are misaligned by 2px (it's a lot easier to fall into these traps than you think).
Some design definitely helps for the demo though. If your hack isn't very UI intensive, I would go with "enough design so it doesn't look like shit."

Engineering Practices
Abstract your code in such a way that you don't interfere with each other. Split up the stack (front end, server side, API's, algorithms, etc) so you can work without dealing with merge conflicts all the time. A good work flow for us is to work separately for a few hours, integrate, test, rinse and repeat.
Do you best to split up your project into smaller features so that if you don't have time to finish you'll at least have something. Avoid what I like to call the "all or nothing project"

Example: if you want to make an online store with a recommendation system and you run out of time for the recommendation system, at least you have an online store.
Example of other situation: We once tried to build an augmented reality game. Division of tasks was good; we split it up into iOS, game AI, computer vision, game flow. We split up tasks well and worked without stepping on each other's feet, but when it came down to it if any of those portions didn't work, the entire project didn't work and that's what happened.

Demo
This is the most important thing if you want to actually do well at the hackathon. A HUGE mistake that almost everyone makes is that they focus too much on implementing features. There have been times where we've been debugging as we were walking up to present.
Instead, maybe about half an hour before hacking ends, stop everything you're working on, even if you're "almost done" implementing a feature. Take out everything that doesn't work and talk about how you're going to present. This is really hard to do mentally but is important if you want to have a functional app by the time you present.

Think about all the work you did and how you can SHOW everyone. If you have some really awesome things going on behind the scenes, figure out a way to display it.

Example: We made an app for the game "Assassins" where you could "kill" someone by swinging your phone at them. Functionally, the only thing that did was update an item in our database and add a line of text in our web app. Now that doesn't look impressive to a viewer. What we ended up doing was making the entire web page flash red when someone died. What the audience saw was my friend swinging a phone at me, killing me, and a website somehow recognizing that and flashing red to show it. Now THAT was impressive.