Electric Vehicle (EV) has become one of the promising solutions to the ever-evolving environmental and energy crisis. The key to the wide adoption of EV is a pervasive charging infrastructure, composed of both the private/home chargers and the public/commercial charging stations. The security of EV charging, however, has not been thoroughly investigated. In this paper, we propose a new class of attacks, ChargeX, which aims to manipulate the charging states or charging rates of EV chargers with the goal of disrupting the charging schedules, causing denial of service (DoS), or degrading the battery performance. ChargeX comprises two types of attacks: (i) Intentional Electromagnetic Interference (IEMI)-based attack leverages the effects of inductive coupling to influence the current readings along the power line, and (ii) cable tampering attack inserts a hardware attack circuit to strategically modify the charging control signals. We design and implement multiple attack systems, and evaluate the attacks on a public charging station and two home chargers using a simulated vehicle load in the lab environment. We further validate the attacks on a Tesla vehicle to showcase the disruptive impacts of ChargeX. If deployed, ChargeX may significantly demolish people's trust in the EV charging infrastructure.