IoT and Big Data Testing: the Hidden Pitfalls
As of the latest Cisco report, there are 27,1 billion networked devices globally, ranging from home appliances to highly specialized machines. All these devices generate a vast amount of data, which is then used to improve technology and enhance UX.
The growing number of smart things proves their actual value and makes companies invest more in IoT development. However, this area has some pitfalls, which are primarily connected with IoT and Big Data testing. This post will dig deeper into the topic and consider how standard software product QA differs from IoT testing and Big Data testing strategy.
IoT vs. common business software testing: key differences
Any software testing means creating new test cases, methods, and scenarios. In IoT testing QA engineers must show exceptional creativity because they deal with unique products that differ in a number of ways:
Variety of platforms and devices. Web and mobile apps usually run in a device-server-database infrastructure. With IoT solutions, things are more complicated. They are implemented through a complex system of interconnected devices and a greater range of hardware, software, network, and client platforms.
Lack of emulators. IoT testers can use device emulators if they don’t have a specific phone or tablet model in place. In IoT software testing, it is hardly possible. Of course, devs resort to some DevKits to try the product at an early development stage, but the final testing should go on real devices.
Hardware tests. Smart devices are an integral part of the IoT system, so they must behave adequately even in non-standard conditions. IoT testers use hard reset, reboot, discharge, fuzzing, and other test types to see how the hardware responds to non-routine events.
Heavy network load. For Internet of Things solutions, fast data transfer is critical. But many involved devices produce a heavy load, making it challenging to achieve the desired outcome. With the various combinations of network equipment, internet speed, and link channels, there is no way to play all possible scenarios. IoT QA experts may use network virtualization to simulate various system states, but this is not the same as real-world testing.
New vulnerabilities. IoT security or penetration testing should come up with innovative techniques because it faces unusual attacks from the “things” side. For example, some time ago, a smart fridge leaked its users’ email logins, and a fitness tracker sent workout info without authentication. In a deep pool of device manufacturers, it is hard to guess where to expect danger from.
Different UX. The look and feel of mobile apps adhere to common ergonomics standards and platform guidelines. The IoT usability does not have generally accepted norms and is often assessed subjectively, which pushes the QA team to rely more on beta testers.
Big Data testing: what QA engineers should be aware of?
Big Data comes from multiple sources and is afterward processed by the Big Data app. It is crucial the latter is reliable and scalable. The Big Data testing approach involves, first of all, checking how the app manages the info:
how high the performance is
how granular the structuring is
how reliable the storage is
Challenges in Big Data testing derive from the endless information flow and the massive data amount often measured in terabytes. To conduct thorough testing QA team checks:
Batch processing – when the array of data comes at the same time, possibly from the same source, or aggregated by some data pre-processor.
Real-time processing – the majority of BigData apps should process the data on the fly, without interruptions.
Interactive processing – in some cases, the data source may expect a response from the BigData app. An example may be a distant meteorological station that gives a data packet with low battery level info and receives instructions to increase the sleep time before the next measurement and communication session.
Five challenges to overcome while testing Big Data
Testers should be prepared to face the following difficulties and know how to overcome them:
1. Implement data filtering.
After the data is collected, you need to filter it out and convert it into a readable format. It can be done through batch processing with the help of scripting or interactively with special cleaning tools.
2. Ensure data completeness.
When retrieving data from one or many sources, you cannot be sure that 100% of the info is captured. To check this, apply Big Data testing tools like Presto and Talend, which help assess the data quality and integrity.
3. Show meaningful results.
Big Data is used to create valuable business reports, but this isn’t easy given extreme information volumes. As a solution, testers may use clustering Big Data into a higher-level view where smaller data collections get visible.
4. Minimize manual tests
Big Data means big amounts of work that are nearly impossible to do manually. QA engineers should be able to produce high-quality auto-tests and develop effective Big Data testing strategies.
5. Set up an appropriate test environment.
Large file sizes set particular demands for testing infrastructure. Testers should get a testing environment with enough storage, processing space, and memory capacity. You may need to provide them with a cluster with a scalable number of nodes.
The best strategy for testing IoT
A testing strategy for IoT and Big Data should encompass methods and cases that prove the system’s reliability, usability, and relevance. It must cover all components that contribute to normal product functioning, namely:
Device
The device can generate data on its own or act as a middle-man transmitting it to third-party services. In either case, testers check if the data is securely protected and efficiently processed according to the prescribed requirements. For that, the following testing types are conducted:
Sensor-protocol compatibility (e.g., using a microphone to record voice data and transmit it via Bluetooth or Wi-Fi).
Device security (checking for possible paths of data leakage and resistance to cyber-attacks).
Device variations (e.g., different models and generations).
Device usability (examining device ergonomics).
Network
IoT devices use different types of connections, such as NFC, Bluetooth, Wi-Fi, LoRaWAN, Cellular, Ethernet, Satellite, etc. The tester’s task is to check the power consumption, range, latency, and bandwidth of the network. They should also test it under different loads and ensure the backup channel is activated when the primary one is not available (if such is provided). Here are essential points to study:
Device-to-device connection
Network security
Network performance
Cloud
Cloud ecosystem reduces the load on internal data centers and storages but brings new challenges in Big Data testing. QA team must ensure all cloud services are properly integrated and device-cloud connectivity goes through secure encrypted protocols.
Cloud testing is a part of the overall IoT penetration testing, which evaluates all the system parts to make sure they are safe enough. Also, cloud elements go through performance and scalability tests as these parameters are critical for smart things operation and Big Data processing.
Application
In the IoT system, a web or mobile app is the closest to the user element. Through the app, the user launches a vacuum cleaner, opens a lock, or peeks in a home or office. App check combines traditional testing approaches with specific IoT and Big Data testing methods. It is vital to verify app usability and intuitiveness. It must be easy to follow, especially if you work with a product of a household nature, such as a smart home or a fitness tracker.
Conclusion
The IoT industry is on the upswing but has not yet reached a mature level. Dealing with “things” and “data,” testers face non-trivial tasks that require unique approaches and methods. The main differences between IoT testing and traditional testing are in a completely new UX and tremendous data amount. Based on these features, testers build an ad hoc testing strategy that scans the IoT system’s entire infrastructure and controls the endless stream of data.
Checkout our case study where Erbis’s QA experts helped win an award during the IoT testing.
