Overview
To understand the role of testing in IoT, let’s have a look at the overall IoT solution architecture. An IoT solution consists of many components and stacks which are depicted in the above figure with vertical line partitions. All these stacks have a software and hardware module that performs a specific functionality. E.g. The sensors in the hardware layer capture data which flows through the gateway WAN, up to the cloud server, where it gets processed and analyzed and then displayed in the user interface. Since each component handles the data in a unique way, the testing requirement for each layer is quite different. This is what defines the testing areas of IoT. Each testing area targets one layer within the entire IoT system architecture and addresses its functionality towards realizing the end-to- end IoT use case. Let’s briefly cover each one of these testing areas.
The very first layer of IoT is the hardware which senses and interact with the physical environment. E.g. it could be a temperature sensor, a wearable heart-rate monitor. Hardware testing involves testing the hardware interfaces, peripherals, connectivity, power handling and other device functionalities. Due to it’s nature, the testing is done mostly manually and involves working with the hardware board with tools such as oscilloscopes, multimeters and signal generators. The software running on this hardware is called embedded firmware. From a software perspective, the focus is more on validating hardware-software interfaces, the accuracy of sensing and actuation mechanism and handling system failure scenarios such as malfunctioning flash or sensor data, software upgrade etc.
The gateway device sits on the edge and gathers data from multiple sensors and sends it to the cloud. The gateway testing requires validation of connectivity with the sensors and the business logic at the edge. Various IoT platforms provide docker based agents which can be deployed on edge devices with a centrally managed way.
The objective of Network simulation testing is to understand the effect of network conditions on the connectivity protocol and test the resiliency of the cloud side software stack against network failure, packet loss and delays. It tests the cloud server functionality against real world situations such as a surge of traffic from gateways, DNS amplification attack, excessive packet losses, frequent disconnections and reconnection attempts. Testing on UDP based protocol is more interesting here as TCP has already a built-in mechanism to provide a reliable service. The testing requires sophisticated tools to model the network conditions and also the volume of traffic coming from hundreds of thousands of sensors.
The cloud platforms are provided by service providers such as AWS, Azure, IBM and are responsible for ingesting and analyzing data at scale. The key criteria for cloud testing are functional correctness of application, along with testing for performance, analytics and scalability. One of the key challenges of testing the cloud applications is to understand the large set of inputs which can be generated by sensor and network conditions. The focus of our documentation will be primarily on this area of IoT testing.
Business applications are hosted on the cloud and are critical to how the user interacts with the system via the web, mobile and other external APIs. These are also responsible for presenting the data to users in a suitable form of visualization to make sure that the users get the correct information about system behavior. Testing of Mobile Apps and end-to-end user experience is also included in this.
Security testing is not an independent area but encompasses all other layers of IoT stack. At the hardware layer, security testing must ensure that there are no exposed open ports or telnet servers which are subjected to brute force attacks. Validating the secure boot, software download functionality and private key management for certificates is also critical. At the network and cloud layer, focus changes to ACLs, firewalls and providing role-based access to various system users.