Computerized monitoring of the home based rehabilitation exercise has many benefits and it has attracted considerable interest among the computer vision community. Nowadays, many rehabilitation systems are proposed, most of the targeted disability is for stroke patient. Some of patient or user just wants to take certain part for rehabilitation. Therefore, this paper is focusing on hand rehabilitation system. The importance of the rehabilitation system is to implement the specific exercise for the specific requirements of the patients that needs rehabilitation therapy. This paper presents the specific hand rehabilitation system using computer vision method. The specific hand rehabilitation implemented in this system is a hand deviation exercise. This exercise is benefited to improve the mobility of the hand and reduce the pain. The hand tracking and finger detection method are used in this hand rehabilitation system. The result of the exercise can be used as a training data for the analysis of the injured hand recovery and healing process.