To support the drastic growth of wireless multimedia services and the requirements of ubiquitous access, numerous wireless infrastructures which consume enormous energy, such as macrocell, small cell, distributed antenna systems and wireless sensor networks, have been deployed. Under the background of environmental protection, improving the energy efficiency(EE) in wireless networks is becoming more and more important. In this paper, an EE enhancement scheme in heterogeneous networks(Het Nets) by using a joint resource allocation approach is proposed. The Het Nets consists of a mix of macrocell and small cells. Firstly, we model this strategic coexistence as a multi-agent system in which decentralized resource management inspired from Reinforcement Learning are devised. Secondly, a Q-learning based joint resource allocation algorithm is designed. Meanwhile, with the consideration of the time-varying channel characteristics, we take the long-term learning reward into account. At last, simulation results show that the proposed decentralized algorithm can approximate to centralized algorithm with low-complexity and obtain high spectral efficiency(SE) in the meantime. To support the drastic growth of wireless multimedia services and the requirements of ubiquitous access, numerous wireless infrastructures which consume enormous energy, such as macrocell, small cell, distributed antenna systems and wireless sensor networks, have been deployed. Under the background of environmental protection, improving the energy efficiency (EE) in wireless networks is becoming more and more important. In this paper, an EE enhancement scheme in heterogeneous networks (Het Nets) by using a joint resource allocation approach is proposed. Firstly, we model this strategic coexistence as a multi-agent system in which decentralized resource management inspired from Reinforcement Learning are devised. Secondly, a Q-learning based joint resource allocation algorithm is designed. Meanwhile, with the consideration of the time-varying channel characteristics, we take the long-term learning reward into account. At last, s imulation results show that the proposed decentralized algorithm can approximate to centralized algorithm with low-complexity and obtain high spectral efficiency (SE) in the meantime.