Features of the construction of tests and software development used in testing the product on assembly lines
You are: the developer of test software for testing the Product, or the manager of these developers, or in general a manager, or an engineer, or a general profile tester ... in general, the person responsible for checking the Product’s output. You have some laboratory testing experience, but no accumulated experience with the factory.
Objective: To competently smooth out the development errors of test software for the factory, made during the product life cycle.
')
Note: Product refers to a device or a part of it with internal logic, SW / FW / whose functionality is to be checked. Example: cell phone, controller, programmable devices, motherboard.
In this article I will share my experience gained over the past 4 years. I hope it will allow you to save extra man-hours and nerves, while increasing the quality of the product.
The note is focused on an experienced developer, tester, technical manager and manager. However, if you lack experience, but you have time, read it, since other people's mistakes can be a valuable source of knowledge.
The information is presented in the form of a large "checklist" of the actions that need to be done, with explanations and examples. Lovers to argue, starting with "but we are not so," I repeat, I share my experience, which helped me and the employer not to lose face.
1. Availability of an interface for testing at the product design stage
If you caught her at the stage of product design and have a voice, inform the management that the presence of any test interface available at the time of the product assembly can make life in the factory much easier. You will immediately receive an objection that any board has test points and contact plugs on some ports. The question you will need to ask is: “Will they be available when the Product is assembled? How to carry out such checks when there is no hole in which you can insert the cord? ”
These simple questions and the answers to them will determine the future of your testing software. Savings on the test interface can result in a bottleneck when testing on the Line (more on this below), which can cost you more than an extra port on the board.
2. Availability of an interface for testing at the iron development stage
Let the interface is in the design. Contact the hardware and software developers of the Product and find out if this interface is in conflict with the rules, regulations and safety standards. For example, the mere existence of a test USB port may deprive Product Certification. Read / Write access through the Serial port can do the same (even if there is only a series of contacts from the port on the board).
Thoughts that ports will not be removed, and you will send test commands through them, call some APIs, can be very dangerous. Communication with the port in the final version of the Product Software may simply be blocked. In the ideal case, it can only be opened to display information. Please do not confuse this with read. Read implies access to specific information. Vigilance does not interfere with the transition from one version of iron to another (P0 -> P1 -> ...), the magic port can also magically disappear. HW management teams and their solutions are not permanent.
3. Machine for testing HW, requirements for it and test iron
It is important to understand that as soon as the intelligible logic will work on the hardware, the hardware developers will begin to create a machine that will be sent to the factory (or already there) to check the HW of the product and its parts. Did you know about this car ? Not? And it exists.
The machine can be an elegant, ready-made solution from a third-party manufacturer or some kind of aluminum-plastic interior design monster containing third-party components and assemblies with a terrible mixture of “samopal”.
Never carp at appearance of the given car , very often it such because of safety requirements on manufacture. A simple voltmeter can be bolted to a concrete slab, only to avoid falling due to vibrations of the conveyor.
During the construction phase of this machine, you will most likely be asked about the equipment you need for the tests. Here it is necessary to have a clear picture of the differences between laboratory tests (one device from the lot is being tested) and production tests (each device is being tested).
The developers of iron and grocery software are paranoid. Requirements for tests in the car HWschiki write themselves. Be sure to go through all and ask questions: “What do you want to check with this test (to cover this requirement)? And why do you want to check it out? And why exactly? ”After a couple of hours of conversation, you will be surprised how 10 requirements will turn into one, or even 0.
By default, they will include everything that is possible in the Line testing documentation. Do not give in. "Why, the test takes only 3 seconds !!!" - "Yes, the test - 3 seconds. Connecting a device for a test of 1 minute, for a total of 63 devices per device, per 100,000 devices it will take 70 man-days. ” Always have a cold and sober calculation of resources in the head or in the form of a table at hand, sometimes it is the only argument in the struggle to exclude or change the test. Make sure your argument is communicated to the interlocutor. Make sure that all agreements are fixed on paper, as the machine will be made and delivered strictly according to the documentation. And if it says there is a test, and you sort of agreed that it is not there, then you are to blame.
Everything you leave in the document will be a production test. Anything you exclude will be a lab test.
Example. The product is radio frequency modules (signal transmission / reception). The HW team puts before you the requirement to check all channels of the 3 radio frequency bands. A simple description of the number of channels (there are a lot of them) and a rough estimate of the time will kill their ardor. Checking all channels is now a lab test (Test 1). Then they will demand to check the boundary channels for each range (this is 2 * 3 = 6 channels). However, all these channels will be covered in “Test 1.” Voila, you now only have the requirement to test the work on the three bands, and this is only 3 channels.
The reader may argue that if we produce radio frequency modules, then we need to check all channels for all devices. The specific answer will depend only on the class and cost of the devices produced. It is enough to compare the costs for the full test cycle for each device and for warranty service of defective devices with the same distribution of defects in the lot, this is bare mathematics. Plus, remember that by the time of mass release, numerous tests will be carried out on all versions of hardware and software (sanity, feature, functional, stress, stability, burn ...), which will repeatedly cover and block all requirements of HW and SW teams.
The second and very important point is the hardware and setup, which will be required of you for the machine to cover the tests. A discussion with HW specialists of specific procedures for meeting the remaining requirements will allow you to determine the range and functionality of the necessary equipment. The mistake that a beginner can make is that you see only one car in front of you or part of it. And it can be more complicated than it will be shown to you, and there may be several of them in the factory. Ask yourself and them questions: “Will the knots inside the machine interfere with each other? Will not the machines standing nearby in the factory interfere with each other? ”Try to take ALL into account in theory - mechanical interference, electromagnetic interference, interference, duplication, wear and so on.
Example. Let's return to the previous one. We need to cover 3 channels from each band. So we need 3 receivers / transmitters, or 1 programmable. Within the framework of one machine there are no problems with implementation. Now put in a row 3 cars , between which there are no screens. What did you get? Interference and interference immediately come to mind. What else? Duplication channels. “But the cars can be tuned to different radio channels, because there are plenty to choose from in the range.” Yes, you can, but imagine the situation. 3 identical machines come to the plant, each comes with a volume of instructions, and they differ only in 2 to 3 lines indicating the frequency of the channels. Will the engineer who compiles and customizes them make any difference? Most likely no. It will be very good if he reads at least one manual completely. As a result, there will be 3 cars on the Line with the same channels: A (1, 2, 3), B (1, 2, 3), C (1, 2, 3). The line starts and the device, which should be on A (1), communicates with C (1). If you think that the engineer will have to blame for this setting, then note that he has the right to change the requirements for the test machine and change it, and he can write off his mistake on this change. For example, three assembly lines (N machines for each) cannot divide the RF range, as it is occupied by other equipment, and setting up each machine for its RF channel may simply not be possible.
Remember that the components and assemblies of the machine used on the line, should test not one device, but thousands. Do not transfer your laboratory experience to the Plant. Develop testing methodology again.
Example. The product is the same as before. Set the task to check the output signal level (power). Steps: we take the device, we connect the test antenna, we measure the output power, we disconnect the test antenna, we connect the factory, we send the device further. Mechanical wear can be critical here. A test antenna must withstand thousands of connections without losing the quality of the signal transmission (otherwise many devices will be rejected). The connector on the Product must also withstand several connections and leave the stock for the future. For many, the discovery that the antenna connectors on our favorite laptops and cell phones are designed for only 10 cycles of connecting / disconnecting the antenna. How many of them do we have? In this test - 2. The device itself can get on the test 3 times (already 6), and then go through a couple of reassembly cycles (already 12). 12 more than 10, then the risk of rejecting the device on the line with your tests is great. So you need to talk with HW people, look for an alternative, or do a laboratory test. And if the “zatyrkannyh” devices only 2%, it is a lot (2000 out of 100,000).
Let the test is still included in the production cycle. What equipment will measure? Even if the "office pays", you should not create a Boeing. Do not transfer your laboratory experience to the Plant. It is best to stay on equipment that will require the least support and can be calibrated on site without being sent to a service center. Even better, if you always have an alternative in stock.
4. Separation of laboratory tests and production
In the previous paragraph, you will already have some test base applicable to the HW test. In the same place their division into laboratory and production will begin.
Let we already have some working and not very raw Grocery software. Then it is time to conduct a dialogue on the verification, the finished device. The tactics are exactly the same, but now we are polling the product software developers and product managers. Questions: “What and how do you want to test at the factory? Why? Why so? What did you use earlier for similar products? Are there any requirements to check the finished device? ".
As a rule, you are not the first mover. Before you there were a lot of people who were engaged in similar work, after them there will be a part of tools, documentation for them and requirements. Examine everything, compare with the responses of developers and managers, highlight the risks, and make a plan of action.
After reviewing everything, it’s a big mistake to tell yourself: “This is all outdated! Curve design! The code is written crookedly! I will do everything from scratch myself! ”. Remember that this old “hogwash with a crooked design and code” has tested a whole lot before you at the factory and / or in the laboratory. So, she (garbage) coped with its task, but you have not.
After reviewing all the documentation and digging into the code of predecessors, make a list of the resulting tests. Try to design it as technical requirements for future test software, much like HW developers did for their machine. The fact is that the standard documentation within the company is very similar. You still have to describe everything for the factory, and so you will already have a draft ready, which you can provide upon request.
Discuss the list with colleagues inside the team. This can be a formal review or a simple conversation. Ask how they would do this or that test. Fix any absurd arguments. As a result, you yourself get a roadmap for almost every test. Naturally their number may increase or decrease.
The final draft break into the laboratory part and the production part. If you have a team, then this partition must be put back on review.
5. Laboratory Tests and Automation
Testers are increasingly tasked with automating tests. The list of vacancies "manual testing" is estimated below the "ability to automate."
Remember one simple thing: “The automatic test covers only the algorithm that is embedded in it. He can never check the step left or right of the algorithm. He will never find an auxiliary problem. 100 hand tests can find 200 problems. 1000 auto-tests can be written in such a way that they will always always pass.
The golden mean - the presence of two types of tests in each phase of testing.
In spite of the above, laboratory checks are best automated. The usual routine test runs of the main branches of the Product SW (feature test, sanity) will still be performed by someone from the team, so there must be a “happy medium”. The laboratory can and should check only the most complex and critical for the product and certification tests.
If you need to continue laboratory testing and prepare tests on the Line, then the laboratory is a great place to roll back the methods of working with test interfaces and automation methods that you plan to use in the factory. The criterion of the “good” method is its stability (100,000 devices must pass the test). But what about the speed of testing? Speed ​​- a nice bonus, which may not be. Recall now the curve design of the ugly code from the last item. Here it is stable. Perhaps his ugliness is a consequence of his stability?
6. Manufacturing tests and factory
Chief engineers, processors (classic QA) and security engineers design the line and make up the documentation. It describes the steps of production on the line. One or several steps will be to use your software (test station, or test line). To add or remove a step from this process, when it is launched, is extremely difficult and almost impossible.
Most Lines are line or snake type conveyors.
Example. Imagine a conveyor belt - a snake that occupies the entire production area (100 people, 3 shifts). There is only a place for evacuation, work of employees and transport lines (supply of components). It is simply impossible to insert a node into this pipeline. He will either not enter (there is no physical place) or will not pass the security check (fire requirements, etc.). If it is necessary to insert the assembly, the conveyor will be completely stopped and, most likely, disassembled. Further, new documentation will be compiled, assembly and configuration will be carried out anew, acceptance and debugging will also take some time, and 300 workers will receive sn as they have received.
"There is a place in the corner, let's put it in there!" You can't even say that. It is impossible to remove a product from a snake, drag it into a corner and then return it to the place in a snake. Confusion and chaos will arise.
At this step, you may be asked to describe the operator’s workstation using your software. Naturally, it will be a PC and a set of necessary test tools and nodes. Barcode reader is a test tool, Ethernet cable is also, USB cable is also so on. Take it seriously and describe everything before the screw, the documentation during the development of the test software can be edited, the more you specify at the beginning, the less you have to add later.
They will also ask you about how the station will work, and what the operator needs to do. And, of course, they will ask for a list of requirements for your software. We recall that you have already prepared a draft in step 4. It will be transferred to the standard documentation templates, subjected to review and returned to you for revision as a list of requirements for test software, but on behalf of the factory. That is, your draft will be returned to you as an approved set of requirements and a plan. Therefore, it is important to exclude from it all the “bad” in steps 1-4.
, .
, 1 1 . , . .
, , , . N , .
, , , . , «» , .
, .
, , .
. printf .
7.
– . . , . review, .
( UI) - .
:
- .
.
It would seem that everything is simple. , , . 500, , open source tool… .
. , Web Python Selenium, Programming Language, Python Curl, , . , , Curl , , ( ). Curl , . « ».
. , , ?
, . .
8.
. . , .
– (badge) . , , ( ) .
ID . – , , .
ID . – , .
« SQLite, ...», . , .
, .
, .
9.
.
. , ? . . ( User Manual) « » . . , .
Why do so? Retry, , -, . .
. 10 , – 144 . 1 , 160. 10 . - .
, - , . .
Example. USB SERIAL. «Test PORT: AUTO / USB AUTO / SERIAL AUTO / SERIAL COM6». . , , - .
, .
Example. , IP c DHCP . USB Linux . IP : PING / ARP, Linux , DHCP API. ( ), USB ( , ), DHCP , , . , .
, – .
10.
- ID, , .
. .
6 . , 3 . UI , .
UI ID ( ) , UI . , UI . . – . ( ) 8 , .
11.
, , .
. , ( 1 2).
, .
1 ( ). USB, USB, .
:
). — USB . ?
). — PC (HW SW) USB . USB hub PC . USB? PC USB touchpad ( USB hub) , ? ( , PC)?
). — USB 2.0, PC USB 3.0 . ( 6) PC USB 2.0. , USB 3.0. , , . workaround, USB 3.0 BIOS. BIOS chipset USB hub . , 2.0 3.0 . BIOS 2.0 3.0. 1 2.0. ? ?
). . , ?
2 ( ). Ethernet , .
. , , Ethernet switch/router, Serial. .
«» . « ?». Need to! , (Arduino, Raspberry Pi). – .
, , ( ).
: « Curl 6 , ?». : «». , . , , HW. , - .
. , – . , . . , 100% . . «? Why do so? ! ». , . . . .
Example. . , ( ). . , , , . 2 ( ) – 2 . , . . , .
, , .
Example. , – , – Ethernet , web . , (a la telnet). : « ?» : « , USB , API, ». API, . API API. – . . , .
. Ethernet.
Ethernet. IP (iperf, ping, wget). . ? Ethernet. Ethernet . , . , : , web – . IE, Firefox, Chrome? 1000 , . . , POST/GET web . , ( 9).
12. , UI
. .
- ). UI . 1 . , . , . .
- ). UI . (, ), , .
- ). , , . . . , . .
- ). UI . , . . « ».
- ). . , . , . . , . . , , .
- ). , . , . , , , . , () . - , , 24 . . : « ?». ? , , « », . : . . , , , . , . UI ( ), , . , .
, .
, . . PASS, FAIL, ERROR… . . , . (11) , – . – . / – . . , . 2 : . – . . , , 500-1000 . - ). UI , , , . , , , . , , UI screenshot . User Manual. , screenshot « » .
- ). . , , .
13. , ,
, . . (), , , ( , xml). . QA , , . 1 2, .
, User Manual. , . ? Xml ?
, ? , . « 3 , ». , ?
: 7 – 13 review . . API – . «» . - API « », . - , .
14. ,
, «» « ». . , . , . (), , . ! ( )! , , , .
, , , ( 12 ). , UI. User Manual.
, . 3 : , , User Manual.
. . . , , . , , ( ) .
. , . . , , , , , .
, , , , – User Manual. . User Manual, , , .
Thus, dryness in the bureaucratic and screenshot parts will minimize changes in them, which are much more difficult to make than changes in the User Manual.
15. Installation package
As mentioned earlier, factory engineers can do a different test station from yours. They can cluster them and / or add something of their own.
Any complex changes may require changing the number of additionally installed programs and packages.
I recommend making the installation package in the form of a clear and not complex script (batch file). With the obligatory indication in the User Manual that this file is not final, and that the user can add or remove necessary lines from it. This small loophole will untie the hands of engineers, and you will be relieved from the need to redo the package provided and, as a result, to complete the acceptance.
An even more flexible approach would be to have several scripts, each of which installs only its part. The corresponding steps in the User Manual should begin with the phrases: “Install, if necessary, such a component using such a batch file”. In this case, any changes to the final installation will be made by adding components, completely excluding changes to your package. That is, your instructions will be taken as a basis, but with Factory changes. The latter will be carried out according to their internal documentation, which will not concern you in any way.
Example. Installation requires: your software, MS Visual C ++ libraries, DHCP server, wget client.
Option A is inflexible. You have an installation exe file. If the Plant decides to change the DHCP server to its own, then you need to carry out the acceptance. It takes a few days of bureaucracy.
Option B is flexible. You have created a batch file that sequentially installs all the packages. You also indicated in the User Manual that this file can be varied. You may be asked to remake the file by replacing or deleting packages in it, as the engineer will not know how to work with batch files. It's not scary, but they can ask for a change every day until they debug. For the final change, the Plant itself will compile the entire package of necessary documents.
Option B is almost perfect. You not only have the main istalation script, but also a bunch of minor ones that install one package at a time. In the User Manual, all the istalation steps are described as optional (except for the necessary ones: your software, MS Visual C ++ libraries), and the “main script” is given only as an example. This option will give even more freedom, since any engineer will be able to exclude or add a step. The disadvantage of this option is a large number of installation steps in the User Manual.
16. Engineering implementation of your test software on the line
At this point, you already ship a stable version of your program to factory engineers. But acceptance will not be carried out yet. Line engineers will test your software in real-time mode. Be sure to ask to note the time of testing the Product (for 1 pc, 5 pcs, 10 pcs, 100 pcs.). The more time statistics you have on your hands, the better. You will need to reduce these statistics to the bottleneck metric and send it to your management prior to acceptance (see step 18)! It is important!
If you have a chance to visit the factory, then it's time to do it. A visit to the Line is sobering and takes you beyond the laboratory.
Remember that on the line you are not authorized to do anything and give advice. All that is required of you is to carefully observe. Even if someone does something wrong, be silent until you are asked. Any of your comments has no authority there, and if it still leads to undesirable consequences, then you are to blame.
Only at meetings, express your ideas and comments.
Any changes that you will need to make, first make on your desktop PC, and then transfer them to the engineer.
If all of you remained in native walls, then prepare for fast "fixes" and changes. Here the flexible design and good program structure will play into your hands. Ask to change the report? You have templates. Ask to add additional ID verification? You already have everything taken into account in the design and there is a place in the UI. Ask to speed up part of the program? Try playing with the parameters that are already in a separate file, you won't even have to rebuild. Want to change the number and order of tests? And this has already been done.
Try not to butt too much with the engineers on the line due to their changes. They work with what is, and with what was laid out at the requirements stage (steps 6-7). They will not be able to urgently buy something or change. Just try to incorporate everything into your software. Complicated? Yes, it is not easy, but if you have implemented additional workarounds (the example of USB and Serial), then there is nothing to worry about you, by and large.
If, however, the problem with innovations has arisen, then try to solve it or bypass it within the framework of the existing equipment. At the engineering implementation stage, you will not be able to say “buy this and that,” they have already bought everything.
17. The introduction of test software on the line
Formally, everything is already implemented there. At this stage, acceptance documents are added and signed.
At this step, your signature matters. She puts a burden of responsibility on your shoulders. Remember that dozens or even hundreds of people on the line will work with what you gave them. The quality of the output product and its parts now depends on your software. Your reputation is now in sight.
18. Bottleneck and third-party problems
What is forgotten even at the Plant is that the Line at the very beginning does not work at full capacity. Over time, and new orders for the product line will then accelerate, then slow down.
The time statistic of step 14 will help you estimate the throughput of your node. This is a very important parameter. The node should not lag behind the rest of the line. If collectors collect 1000 Products per shift, then your site must have time to test 1000. Make a “bottleneck” calculation. This is a simple mathematical approximation of the number of devices tested per unit of time. In fact, engineers on the line do such calculations always, which is why they combine test stations in clusters. Their experience, however, is different from yours, and they can “calculate” the ideal case, so you need to duplicate their work by doing the way you understand it.
In the event of a “neck” on any node of the Line, it will be stopped until “the balance of forces is resumed”. If the line is not stopped, there may be overproduction (who will pay for it?), Clutter of warehouses and temporary storage facilities on the Line (may conflict with fire safety), confusion (it’s not possible to tell parts of the Line’s staff to stop to accelerate, it will be much more difficult to give a return order).
As a rule, the “neck” is a Line problem, but if you have questions, then you already have a report sent to the management with this statistics. In this case, any questions "change the tone" and become just one: "Can you do something?". The answer, most likely, will be hidden in the parameters that you carried out in advance in a separate file.
On the Line, there may also be third-party problems that no one has foreseen at all. Usually they are solved promptly. But a number of them may be cumulative. To avoid “critical mass” on your site, ask for a step in the process that will require the engineer to send you the entire set of logs from the line once a day / week — these are your reports and the engineering log of step 12 E. Analyzing this data will allow you quickly identify problems missed at all stages of testing and implementation.
Source: https://habr.com/ru/post/316332/
All Articles