I tried to put this into getting around, but it didn't work, but trains and bikes are off the beaten path.|
Train - only Auckland and Wellington have local trains. TranzScenic has daily trains Auckland - Wellington (weekends only in winter), Wellington - (ferry) - Picton - Christchurch (suspended till 15/8/2011) and Christchurch - Greymouth. They are usually slower than buses, but more comfortable.
Bicycle - most cities have started on providing cycle lanes. Most roads have hard shoulders wide enough to cycle in. Repair shops are often a long way apart. Most of the long distance walkway can be cycled - http://www.teararoa.org.nz/index.cfm