Deleting custom fields

I have a template where I have used "Add new field" and dragged it into the template to create the custom field. I no longer require them now but when drgaging them back to the left of the screen where the list of fields are, there is no way to delete them.