Like the others have said, go with what you like! It's your bike and your opinion is what counts. If you think the bike looks sportier without the top box and if you want a sporty looking bike, don't leave it on. I like bikes that look like they're going long and far, so I like the way my bike looks with a top box on.
When I got my first new bike, it was a '99 Kawasaki Concours, I liked the way it looked without the top box. My wife rode with me then and to carry stuff for two of us, we needed a top box (plus it gave her a backrest that greatly increased her comfort and sense of security). I figured I'd leave it off unless we were traveling together. But it turned out that I liked carrying somethings in the top box and other things in the Connie's panniers. I thought about taking the panniers off unless we were traveling, but I didn't like the way the Connie looked without them.
So, for me, the top box stays on. And, of course, with the NT, the panniers don't come off.
It's up to you!