Again, I want to apologize for not finishing this series before now, but the reason that I even started the series was because we were executing a project at a steel mill that would involve a heavy amount of simulation and validation. I also had personally been having conversations with customers and our management team around the topic since I have been with LSI. Doing proper simulation and factory acceptance testing here at LSI has long been a part of our culture as we have found that it reduces time spent on-site and makes the startup and commissioning phase of the project go much faster and much smoother.
During the steel mill project that I mentioned above, we wrote simulation code against the functional descriptions that we had written and had agreed upon with the customer. We then had their engineering personnel come to our panel shop for an integrated FAT with hardware, servers, computers, and all software running before we invited the production staff to the actual FAT. This way, we insured that the application was as we (the engineers) all felt like it should be before the production people showed up.
After the FAT with the engineering group, the production staff came to the actual FAT and they went through all of the HMI screens, asked questions, and verified functionality of everything. Through this process, a few things were discovered that were missed or needed to be added; and a few bugs were found, identified, and corrected. This allowed all of us to feel very confident that the project would be successful once all of the hardware changes were made and all of the I/O was checked out.
Next, after taking delivery of the panels and other hardware, the customer set all of the panels in place next to the existing cabinets (not always a luxury that everyone has) and all the power and networking connections were made to all of the cabinets. The new server and computers were installed in the pulpit, and the system was run in parallel or “shadowed” for several weeks as operators became familiar with the screens and gained confidence that everything would operate correctly. This also further validated the application that LSI wrote against the running system.
During the outage, the hardware was installed, I/O was checked out, and the system was given a dry run several times to ensure that the system operated as intended. The actual startup (post production support) phase required our engineers to be on-site around the clock for 7 days. Our engineers ended up leaving after 3 as we had all of the as-built documentation completed while we were on-site and the system was running trouble free. The only time that we were summoned for help was one of our engineers was found by one of the steel mill’s electricians to come get a steak that they were grilling in celebration of the end of a long, successful week. One line of code was changed after production started and the customer changed it because they had told us to take the line of code off scan because it wasn’t needed. It turned out that it was required, and they didn’t have the heart to make us change it again.
I must say that I made some personal observations that I think would prove invaluable for any controls engineering project:
- Specification and up front design is critical for success. This design work has to be agreed upon by both parties (customer and engineer(s) performing the work)
- Simulation is key to validating control system software
- Factory acceptance testing ensures that operators, maintenance technicians, and engineers all have a chance to work through any potential issues in enough time before startup such that changes to the application can be made prior to startup. Realistically, changes should be minor at this point in the project if proper specification work was completed
- Shadowing can be an effective tool to promote acceptance of a new system if it is feasible to do (i.e. if there is physical room to set the new equipment)
- Customer engagement through the project is essential. The steel mill’s engineers were at our site about every 3-6 weeks during the major design and implementation phases of the project to check our progress, make changes, and validate our effort. This was a critical stage in the project where so many customers just assume that the work is getting done the way it needs to be done
- Outage planning is critical. On the project detailed in this blog, the engineering team at the mill, contracted electricians, and our engineers came up with a great plan jointly, were staffed appropriately, and the installation was actually completed about a day and a half ahead of schedule, which was hugely important in being able to make the dry runs before startup
I would love to hear back from others in the business about their experiences with the approach outlined in this series to see what your thoughts are. Please feel free to share and to comment!!