People using their mobile phone to browse the Internet are usually looking for information "here and now". Browsing a website that is designed for a desktop computer gives your customer a bad experience and most likely will result in the customer leaving your site. In many countries it is quite common to have 20% of all browsing traffic coming from mobile devices. This is a large amount of visitors you will lose out on unless your website is adjusted to the small screens.
Another fact that also is important to notice is that websites that are optimized for mobile devices, also get better ranking in Google when searching from a mobile device. Google is thereby directing their visitors into websites they know will give a better experience. You should take advantage of this extra traffic by upgrading your website to be mobile friendly.