Modern authentication systems still suffer of some limitations that threaten users’ data protection. To strengthen the overall security of authentication processes, the continuous authentication paradigm has been increasingly employed. Nonetheless, this fresh methodology still poses a number of challenges that remain unsolved. In this paper, we present a novel framework that is able to provide context-aware IoT-based continuous and non-intrusive authentication and authorization services. To do so, we propose a collection of ontologies that represents the defined information model. Hence, these ontologies are combined together with efficient authentication and authorization policies to build a full-fledged IoT Continuous Authentication Framework (IoTCAF). The conducted experiments demonstrate the feasibility and scalability of the proposed framework leveraging the characteristics of IoT pervasiveness.