The similarity metric in Loop closure detection (LCD) is still considered in an old fashioned way, i.e. to pre-define a fixed distance function, leading to a limited performance. This paper proposes a general framework named LRN-LCD, i.e. a Lightweight relation network for LCD, which combines the feature extraction module and similarity metric module into a simple and lightweight network. The LRN-LCD, an end-to-end framework, can learn a non-linear deep similarity metric to detect loop closures from different scenes. Moreover, the LRN-LCD supports image sequences as input to speed up the similarity metric in real-time applications. Extensive experiments on several open datasets illustrate that LRN-LCD is more robust to strong condition variations and viewpoint variations than the mainstream methods.