The problem of extracting an object from an image, known as "image matting," has been deeply investigated in the last two decades. Its extension to address video, i.e., "video matting," has gained less attention. This paper studies the difficulties that should be addressed when extending image matting methods to work on video. In particular we address the following problems: (i) While the required user intervention is reduced over the years, it still remains unpractical to manually provide user marks on every frame; (ii) the temporal continuity of a video sequence suggests that it may be beneficial to solve the matting equations for the entire video simultaneously. This, however, is impractical due to memory constraints; (iii) because of the huge size of input data, interactive results update is not trivially plausible. To cope with these difficulties, we exploit video continuity. Temporal continuity is used to propagate user provided hints, while spatial continuity is used to reduce the number of unknowns and the number of constraints. Finally, we apply an incremental matrix factorization to allow real-time insertion of new user-provided constraints.