Plastic pieces that are located inside a water cooler are often the culprits, causing the water to have a plastic taste. It's a good practice to flush out the cooler thoroughly before using it, as the process can remove any dust or dirt that may have accumulated as well as the plastic taste that is associated with new plastic containers.