"IOT sensors use a variety of technologies to collect information, communicate with each other, and transmit data to computer systems that store and process data.
In typical applications, IOT devices wake up regularly, measure, and make judgments and decisions based on the data of each sensor. It also communicates with other devices and may send data to the back-end system in the cloud for processing.
For example, a medical health tracking device requires different types of sensors to monitor various body parameters, and then process these data parameters through the system or cloud system carried by the device to generate reports and send effective information to medical personnel.
Then, the doctor can start corresponding medical measures, such as drug intake, according to the data sent by the terminal monitoring equipment. Computers and mobile devices can connect to medical devices through the Internet and remotely manage terminal devices to trigger this behavior. Some systems even include advanced applications that can monitor fluid levels in patients and automatically send information to medical personnel if they are below a set threshold.
Due to the mixing of different devices, platforms, protocols and communication gateways, IOT systems often encounter potential performance, reliability, interoperability and scalability problems.
Unfortunately, it is not easy and challenging to test the performance and reliability of each part of the Internet of things system.
The problem of difficult testing has affected the smooth internal process of the Internet of things system: in this paper, we will study effective and simple testing methods.
What is tested in the Internet of things system and why
With the rapid growth of the global Internet of things market, manufacturers are stepping up the production of a variety of low-cost sensors. In order to reduce the cost of sensors, some sensor manufacturers directly ignore or sacrifice some functions of sensors, which makes many Internet of things devices vulnerable to multiple security, performance and reliability problems.
Typical limitations of IOT sensors include small physical size, low power consumption, small amount of memory and limited processing power. This means that you need to minimize the number of processing processes, so minimize the size of your application. When designers try to deal with these limitations, they can omit key steps and simplify the process, which will expose a variety of technical vulnerabilities in sensors and Internet of things devices.
Testing the benefits of IOT devices
In order to provide effective value to customers, testing equipment and networks is very important to verify whether they work according to specifications. This will enable developers to quickly identify and solve problems or bugs that may affect performance, reliability or user experience.
Block diagram and related challenges of RF module of Internet of things
In addition, the test ensures that the Internet of things system complies with security and other standards. For example, a drug needs to be approved by the FDA before entering the U.S. market. Internet of things devices, especially batteries, must meet the performance and safety standards specified by regulatory agencies such as UL (Underwriters Laboratories). Other regulations ensure that the equipment will not exceed the specified carbon emission standards.
If the product does not comply with the existing regulations and standards, it will bring more problems to users and manufacturers. For example, in 2017, FDA had to ask the manufacturer to recall 500000 defective medical devices after realizing that St. Jude Medical RF pacemaker had potential safety hazards,
A test shows that hackers can remotely access the device through vulnerabilities and reprogram it, which will make the working state of the device unstable, even damage the battery and endanger the personal safety of users. Although the FDA did not require manufacturers to recall these vulnerable devices from the market, it approved a firmware update to solve the vulnerability and required users to claim compensation from health device suppliers.
Considering other benefits,
By finding out the equipment defects, it provides an opportunity to solve the performance, interoperability, security and security problems. For example, security defects, such as sending unencrypted data, will lead criminals to easily steal sensitive private information. Hackers obtain relevant sensitive live information by accessing and viewing the surveillance camera. If hackers sell these video information to relevant stakeholders, It will bring great risks to equipment users.
Accelerate time to market.
Improve the user experience by ensuring that users can operate seamlessly on various devices and platforms,
A fully functional IOT test system includes sensors, connectivity, mobile applications, environment and network interface. However, the specific details of the test system test will vary according to the application and system design.
Through a large number of test cases, the best way to understand. What to test, why and what to test. Let's see what factors you need to test in detail.
performance testing
Performance testing will study how devices, networks, software, internal and external applications function to achieve the purpose of Internet of things devices. The performance test determines a large number of parameters, including execution flow, timing, response time, reliability and stability.
The performance test also checks the availability, customer satisfaction and the degree to which the IOT system meets the SLA. In addition, the test can speculate and determine how adding more equipment and workload will affect system performance, responsiveness and stability.
Performance testing depends on three main levels:
System level: processing, analysis, database, etc.
Application level
Network and gateway level: Test Wi Fi, Bluetooth, Z-Wave, RFID, NFC and other technologies, as well as specific protocols such as HTTP, COAP, mqtt and other Internet of things
For example, testing can determine the amount of computation required for embedded software to perform or the ability of the communication network to be reliable and stable under certain operating conditions.
During performance testing, it is also important to determine conditions such as power consumption, temperature changes, memory usage, and other variables that affect or are affected by the process.
Safety test
Each node in the Internet of things system is a potential entry point for hackers. If hackers invade, the performance of a single device or even a group of connected devices may be greatly reduced.
Nodes vulnerable to hacker attacks include applications that collect data, sensor networks, communication protocols between devices, relay device interfaces, and so on.
The researchers showed several security vulnerabilities that allow criminals to access Internet of things devices and steal data or remotely operate Internet of things systems.
For example, hackers can intercept data from medical devices and maliciously modify them, making the information sent to doctors inaccurate. Then, without knowing it, doctors increase or reduce the dosage or type of medication according to the wrong information, which will seriously endanger the life safety of patients.
Similarly, hackers can change the dose of the automatic drug distribution system. Hackers can also use technical means to drain the battery of key devices such as cardiac pacemakers - which will put patients at greater risk of life.
In a test demonstration of the security conference, the researchers demonstrated to the participants how hackers attacked the wheelchair Internet of things equipment, making the wheelchair lose the function of security protection and causing great security risks to users.
Not only in the medical industry, these security vulnerabilities span almost all industries. The test also shows how hackers access the vehicle's navigation data and change its control system data, including taking over the driving and braking systems.
The security test mainly verifies the security of devices, data, applications and related networks. The test focuses on device authentication, device level data transmission protection and server data protection.
Detecting the security risks of the Internet of things at the device and protocol level helps to identify hidden system vulnerabilities and prevent the whole Internet of things system from being attacked at the source. At the same time, it also verifies the effectiveness of encryption and decryption to ensure the security in the process of data transmission.
Usability test
Usability test verifies whether the equipment is easy to use, whether it is suitable for a certain application, and can provide clear results while being easy to use. The test focuses on the end user experience, including the functional implementation of all features of the device. Factors considered include portability and the ability to normally receive and send notifications and error reports when the device encounters problems.
According to the device design, the availability test verifies its ability to collect and process information in the Internet of things device or cloud system. It also ensures that it can display clear information on mobile devices or computers.
Usability is best tested by users themselves. The after-sales support team may put forward some queries such as questionnaires, and real users can feed back their use experience and expected functions. However, the main usability tests are completed by designers, developers, quality control personnel, marketing personnel, customer service personnel and others within the company.
Both marketing personnel and customer service personnel care about what customers need. However, customer service personnel care more about the problems encountered by users in the process of use.
Network and connectivity testing
Seamless connection is very important for the Internet of things, which ensures that all devices of the whole Internet of things system can interconnect with each other and keep in touch with other systems (such as Internet of things server).
The network performance test checks the connection and data transmission speed between the equipment and different network layers. This helps identify and solve problems such as latency, bandwidth bottlenecks, or packet loss that may disrupt connections between devices or networks.
Connectivity testing is performed when the network is started and disconnected. Here are the differences between the two tests:
Online testing focuses on the connection between devices and applications, data transmission and network security.
The offline state tests what happens when the network is unavailable. Especially in key devices, such as cardiac pacemakers or health monitoring devices. Regardless of the status of the network, the operation of these devices shall not be affected. The device must have the ability to store or process the collected data while offline, and then transmit it when the network is restored.
Internet of things device compatibility test
This test is performed at the application and network layers. It tests the compatibility between different types and versions of hardware, software and networks. These tests can determine whether there are performance problems in the combination of some IOT devices, networks, mobile devices, platforms, operating systems and environments.
Device interoperability
This will evaluate how to combine various devices using different interfaces, network topologies and protocols, and allow data exchange across platforms and devices.
Regardless of the brand and model of the user interface device, the type and version of the operating system, this test will verify whether the systems are well compatible with each other and generate the desired results in a standard format accessible to users and applications.
Rules and regulations
IOT devices need to comply with a variety of regulations and access standards, depending on the application direction. The test ensures that all equipment is safe to use and can withstand the corresponding EMI level and other required standards. This test should find out whether the equipment radiates excessive interference signals and whether it can withstand external interference signals.
Like other IOT components, batteries must be safe, reliable and meet safety standards to ensure the reliability of IOT devices and reduce the risk of failure, which is very important for medical and other sensitive application devices.
In particular, the conformance test aims to test the discharge and radiation levels, corrosion and performance characteristics of the battery under different load conditions and even beyond the rated conditions throughout its life.
The current waveform analyzer is suitable for ultra-low power IOT devices as low as 100Pa
Challenges of testing IOT systems
Testing IOT systems is often a challenge due to the need to consider the mixed use of standard and non-standard hardware and software technologies. With the increase of the number of equipment, the difficulty of application and network increases with the establishment of appropriate environment and the selection of appropriate test tools.
Unfortunately, the traditional test process of powering up one or more input triggers and verifying the output results is not applicable to IOT systems.
Simulating all possible test environments is complex and sometimes impractical. For example, unreliable networks, complex real-time scenarios, different device states, sensor data and unstable devices will produce almost countless combinations of test scenarios.
Other challenges include third-party components and subsystems, uneven quality hardware, security and privacy issues.
For example, about 20000 devices send data per second using various protocols. In this case, it is almost impossible to test all possible combinations.
Testing a large-scale system that has grown over a few years will become more complex. In this case, aging devices may only use earlier protocols (such as COAP) for communication, while newer devices use HTTP, mqtt and other modern technologies. This makes it necessary to replace the corresponding test equipment for different protocols in the test process.
Let's look at what other challenges testers will face.
Complex hardware software grid
The lack of standardization leads to the cluster and mixing of multiple technologies in the Internet of things ecosystem. These Internet of things systems may be composed of different types and versions of hardware and software technologies, and the system nodes are expected to communicate with each other in real time or near real time.
Components and subsystems from different manufacturers and third-party providers may use incompatible technologies. When one equipment fails, it may affect the test process. In addition, the quality of the sensor may not be high enough to provide accurate test results.
Testing such a grid is a complex process, which requires proper planning, testing equipment, facilities, expertise and capabilities, which may be unbearable for some QA teams.
A large number of test scenarios
A comprehensive test should verify how the equipment and system work properly, as well as the unreliable or weak network, low battery power of sensors, security problems, packet loss and other actual conditions.
When testing thousands or millions of devices and applications on different networks, it is difficult and expensive to build a dynamic environment similar to Internet of things devices running in real application scenarios, which is an extremely complex process.
Due to the wide variety of equipment using different technologies, it is almost impossible to simulate this real-world load condition. There may also be some unpredictable conditional factors, or this includes the number of concurrent users, the reliability of user equipment and Internet connection, etc. These are all factors that testers cannot simulate the test environment, making some tests impossible.
Another challenge is to determine the consumption of battery power in different operation modes from standby mode to full operation mode.
Scalable problem
Generally speaking, the number of IOT sensors increases exponentially. Although the initial amount of equipment installation is small, it will eventually grow to thousands or even millions in a few years. With the increase of the number, the test tools and scripts designed for thousands of devices have become obsolete and can not handle the deployment of such large-scale IOT devices. In addition, it will become more difficult, slow, expensive and error prone to run test cases and analysis results. In this way, the process of finding fault codes or equipment is very painful and arduous.
Different communication protocols
Internet of things devices use a variety of connection protocols, but there is no standardization. Generally, protocols include HTTP, mqtt, COAP, XMPP, DDS, AMQP, alljoyn, iotivity, etc. These are usually different, some do not follow common standards.
Although some have stagnated, others are still developing, evolving and changing when they mature. Due to the Internet of things
Our other product:
