Youthful Exuberance in Engineering

It is well documented that there is a talent shortage in the engineering ranks and especially in controls engineering these days. It seems as though we are losing more engineers than we are gaining in our industry and it is a shame. The demand has never been greater and almost every customer I talk with has a similar need and similar struggle in finding good talent. We have struggled to find qualified engineers who would fit our culture out in industry and even when we do find good ones, it often takes them a while to get used to our culture and our pace in many cases if they have come from a plant environment.

About 6 or 7 years ago, LSI hired several young engineers and began training them as we have found that we needed a new generation of engineers and leaders for our company. We lost a few, but kept most of them, and many are the bright young stars at our company. We have recently had another hiring spree of young engineers because of our growth (our business and number of employees has doubled since 2008). However, the challenge is how do you train these young minds when you are busy? I talk with customers frequently and they have the same challenge, so we are not alone. I have asked how they are doing it and it seems as though training is simply “trial by fire” with little guidance is prevalent today because every company is running lean and they don’t have the training budgets nor the time to be able to train and mentor young engineers.

We at LSI are facing a similar challenge. We have to find billable work for these guys to work on yet they have limited skills at this point (as they have almost no practical experience from their schooling), and we can get too busy to take the time to train them. So, we have implemented several levels of training to try to get the “newbies” up to speed and give them a good basis for doing their work. We are training once a week at different times (early morning and late afternoon) and at various levels (from entry-level to intermediate) on various topics, along with the general work assignments they are given. we are also assigning “mentors” to each young engineer to guide him or her through their career here. Even with what we are doing, it is still a challenge.

I still remember being their age and all I knew about PLC’s was the following:

1. I knew how to spell PLC
2. Some guy named Allen Bradley was really famous in the industry and must be rich!

Through all of this, I have been so impressed with their willingness to dig in and work. We have been blessed with a tremendous group of young engineers who have a personality and a work ethic to go along with a degree. It is so encouraging because you keep hearing how this generation has a sense of entitlement and is lazy, which at times is true from my experience, but we have been blessed with young people who are humble and hungry to learn. Feeding their minds with all of our collective experience while we all remain productive is going to be a challenge. I would love to hear how other companies are meeting this challenge of finding, attracting, keeping, and training talent. Please comment or send me a private note!

PLC’s Versus PC’s For Control – Is There Still a Debate?

Controldesign.com is revisiting an article that they published in February 2001 where they debated PLC’s versus PC’s in industrial settings. I saw the latest update on this topic today. I remember the debate raging with several customers when I went to work for Rockwell Automation as a sales engineer in the fall of 2001. I remember Steeplechase Software (now a part of Phoenix Contact) having a marketing video with a PC on fire and the machine still running as a show of robustness. I could only think to myself that if the PC was on fire and I had no way of shutting the machine down, then I wasn’t convinced that was a good thing that the machine was still running. Rockwell of course, had a “me too” product with SoftLogix and I remember selling one of them on a test system at a plant when they were replacing a homegrown VB based test system that had integrated LabView and an SLC500 processor.

We at LSI actually sell and use PC based, PLC based (PAC also if you are into that acronym), and DCS based controls and we are of the opinion each has its own merits. When we are selling our test systems (DALS Solutions), PC based control is the only way to go and we actually talk customers out of a PLC based solution because of speed, configurability, and database integration. PLC’s just don’t make sense. We also believe that there are many more mission critical tasks that PC’s just have no business trying to control. It is simply a matter of picking the right tool for the right job. Why would you use a screwdriver to hammer a nail?

The right tool for the right job is critical for any job you are doing whether it is a controls project or just something around the house.

I also take issue with a few statements made in the article as I think there have been some advancements over the last 10 years that the author has overlooked. Note the statement below:

PLC proponents have long claimed that PC-based systems running a Windows operating system (OS) are not a good solution for real-time operations because a PC is not deterministic, i.e., you cannot guarantee a response time to interrupts and tasks. A PLC, on the other hand, has a guaranteed scan time, ensuring its I/O and interrupts are serviced on a regular basis.

Recent developments in Windows and other operating systems may be close to making this a non-issue. Even worse for PLC manufacturers, some people think that PCs are now more responsive for real-time control applications than PLCs. Isn’t that heresy?

This stems from the PLC’s penchant for processing relay ladder logic sequentially. Historically, a PLC started at the top of a list of logic. It refreshed its I/O, and then ran through the entire sequence from top to bottom. This is called a scan time, and it is fixed. Many PLCs still run this way; others may use different languages such as Function Block or Flowchart to avoid running all the logic.

In a PC, scan time is not a limiting function. A PC’s non-ladder logic control programs are often based on interrupt service and changes in I/O, so they don’t have to run through all the logic every scan cycle. Instead, they deal only with those portions of the software affected by the change in I/O. They can accomplish more in the same processing time because they aren’t evaluating all the logic all the time.

If you look at just the ControlLogix system, you can do interrupts, prioritize logic to be scanned based on a deterministic rate, or do traditional continuous scans as mentioned. That is all part of the design of the system and one that we take advantage of every day.

Allen Bradley ControlLogix

I know that many other processors have the same ability in the marketplace today. For that matter, you had selectable time interrupts many years ago in the PLC5 product line. So, I think that there needs to be some clarification added to the article.

I do, however, agree that there is some convergence between the technologies with solid state hard drives in PC’s; prevalent Ethernet communications on PLC’s; Webservers, Historians, and Databases on solid state devices in PLC racks and the like.

I don’t think that the situation today is a tossup as the author concludes as I see very little PC based control on the plant floor, except in highly specialized applications like test systems. The technology has converged some in the last 15 years and I am curious as to how the debate will change over the next 15. Will we see tablets on the plant floor? Will they replace PC’s and laptops? It is hard to say, but I know one thing, technology change is here to stay. Get used to it.

Eliminating the White Knuckle ride of Startup: The importance of Simulation and Factory Acceptance Testing, Part III

Again, I want to apologize for not finishing this series before now, but the reason that I even started the series was because we were executing a project at a steel mill that would involve a heavy amount of simulation and validation. I also had personally been having conversations with customers and our management team around the topic since I have been with LSI. Doing proper simulation and factory acceptance testing here at LSI has long been a part of our culture as we have found that it reduces time spent on-site and makes the startup and commissioning phase of the project go much faster and much smoother.

During the steel mill project that I mentioned above, we wrote simulation code against the functional descriptions that we had written and had agreed upon with the customer. We then had their engineering personnel come to our panel shop for an integrated FAT with hardware, servers, computers, and all software running before we invited the production staff to the actual FAT. This way, we insured that the application was as we (the engineers) all felt like it should be before the production people showed up.

After the FAT with the engineering group, the production staff came to the actual FAT and they went through all of the HMI screens, asked questions, and verified functionality of everything. Through this process, a few things were discovered that were missed or needed to be added; and a few bugs were found, identified, and corrected. This allowed all of us to feel very confident that the project would be successful once all of the hardware changes were made and all of the I/O was checked out.

Next, after taking delivery of the panels and other hardware, the customer set all of the panels in place next to the existing cabinets (not always a luxury that everyone has) and all the power and networking connections were made to all of the cabinets. The new server and computers were installed in the pulpit, and the system was run in parallel or “shadowed” for several weeks as operators became familiar with the screens and gained confidence that everything would operate correctly. This also further validated the application that LSI wrote against the running system.

During the outage, the hardware was installed, I/O was checked out, and the system was given a dry run several times to ensure that the system operated as intended. The actual startup (post production support) phase required our engineers to be on-site around the clock for 7 days. Our engineers ended up leaving after 3 as we had all of the as-built documentation completed while we were on-site and the system was running trouble free. The only time that we were summoned for help was one of our engineers was found by one of the steel mill’s electricians to come get a steak that they were grilling in celebration of the end of a long, successful week. One line of code was changed after production started and the customer changed it because they had told us to take the line of code off scan because it wasn’t needed. It turned out that it was required, and they didn’t have the heart to make us change it again.

I must say that I made some personal observations that I think would prove invaluable for any controls engineering project:

• Specification and up front design is critical for success. This design work has to be agreed upon by both parties (customer and engineer(s) performing the work)
• Simulation is key to validating control system software
• Factory acceptance testing ensures that operators, maintenance technicians, and engineers all have a chance to work through any potential issues in enough time before startup such that changes to the application can be made prior to startup. Realistically, changes should be minor at this point in the project if proper specification work was completed
• Shadowing can be an effective tool to promote acceptance of a new system if it is feasible to do (i.e. if there is physical room to set the new equipment)
• Customer engagement through the project is essential. The steel mill’s engineers were at our site about every 3-6 weeks during the major design and implementation phases of the project to check our progress, make changes, and validate our effort. This was a critical stage in the project where so many customers just assume that the work is getting done the way it needs to be done
• Outage planning is critical. On the project detailed in this blog, the engineering team at the mill, contracted electricians, and our engineers came up with a great plan jointly, were staffed appropriately, and the installation was actually completed about a day and a half ahead of schedule, which was hugely important in being able to make the dry runs before startup

I would love to hear back from others in the business about their experiences with the approach outlined in this series to see what your thoughts are. Please feel free to share and to comment!!

Manufacturing Homecoming?

First of all, let me apologize for my/our lack of regular posting in a while. I lost focus and realize that this needs to be a regular event if we are going to make a run in social media. Next week, I will post Part III of eleiminating the white knuckle ride of startup, but I happened to run across something today that caught my eye.

I have been reading the daily email update for manufacturing.net (called “Today in Manufacturing”) for several years now and several times a week, I click through to an article that they have put a snippet of in my email. The one today on “A Homecoming for US manufacturing?” caught my eye.

Our business has exploded in the last 5 years and since our core is in manufacturing, I was interested to see what the article said because my feeling is that we are seeing a swing of the pendulum of manufacturing back to the US.

I even mentioned in my PILOT presentation a few months back to the city of Memphis that one of the reasons for our growth was a reshoring of American manufacturing. We at LSI also fully believe that the right kind of manufacturing is coming back – the type that requires more skilled labor and higher automation content for all of the reasons that the manufacturing.net article points out. What do you think?

US Manufacturing has been on the rise in recent years

LSI launches DALS website

05/07/12 – LSI has recently launched a new website for its DALS product. This is a set of software and hardware tools that is geared at the quality control and testing industry which has primarily been used in refrigeration and air conditioning test systems. The old suite of products was called “MQCS,” but DALS (Diagnostic Assembly Line Systems) is more appropriate, as the principles applied could apply to an assembly line, not just with refrigeration and air conditioning testing.

 

Check out the new website here:

http://www.dals-solutions.com/

What’s new at LSI – July 25, 2011

I don’t usually like to use the blog space for “bragging” or for “commercials,” but the last 3-6 months at LSI have been pretty exciting with bookings and with projects completed. I thought I would list some accomplishments that have come across my desk:

• Delivered a major system upgrade out of the Golden office (to RAVE reviews from the customer – I cannot stress enough how pleased the customer was) to a major beer producer that included:
  • Malt delivery system cutover from relay logic and Norpak controls to DeltaV in a 48 hour period
  • Cut over the Batch and Grind and Water Blending Systems in 7 days
  • Also performed a major PLC/HMI upgrade to the grain handling system that was on an original Allen Bradley PLC (i.e. predecessor to the PLC-2). LSI upgraded the system to Controllogix/Intellution
• Completed a Digester upgrade to a DeltaV DCS in a liner board paper mill. This effort was deemed a major success by all parties
• Completed a Drives and Motors upgrade on a liner board paper machine from DC drives and motors to AC liquid cooled drives and motors. LSI provided Installation engineering, checkout, mechanical installation support, and PLC programming support. LSI partnered with Rockwell Automation to give the client a major success
• Provided a 692 drawing IFC package in 6 weeks to a major utility to rave reviews, as here is the client’s words: “All I can say is: My hat off to all of you at LSI that made this level of detail in such short time period.  I still need some time to digest the material but what I can see is that this is world class work and I truly appreciate having you as a most valued team member for controls and instrumentation work we will see now and in the future.” And….. “You guys are all amazing and true professionals, you make my life a lot easier so thank you very much for all the good work you keep doing for us.”
  
• Completed a $700,000 panel (102 panels total) order for a pipe manufacturer that spanned both panel shops (Memphis and Rossville) with a design that was changing during the procurement and fabrication process as the client had an accelerated project schedule. This project was also done to rave reviews as I talked with the client last week and he stated to me that we did a heck of a job giving the challenges with the design and the short timeframe with which to build them (16 weeks to procure and fabricate all panels – over 2,000 man-hours). LSI also provided controlnet expertise during the panel checkout as the client’s local engineers had little experience with the technology.
  
• Booked a large order at a power plant in China that will involve integrating a Rockwell Automation Factorytalk View SE system to a Controllogix architecture
  
• Received the 2^nd order for DeltaV configuration effort in China at a chemical producer. A US based engineer will be travelling next month for support and commissioning of this project with our China team.
  
• Is executing a project that involves moving a candy making line from one plant to another out of our Rossville, GA office. The effort includes electrical design (panels, conduit and cable schedules, etc.), programming (Controllogix PLC‘s and Iconics HMI), and overall project management. This effort may lead to similar work in China.
  
• Booked a large order for an acrylic manufacturer for a DeltaV upgrade for their site in TX. This comes on the heels of years of successful work at their Memphis location.
  
• Our MQCS group (http://www.logicalsysinc.com/industries/69-refrigeration.html and https://logicalsysinc.wordpress.com/2011/01/20/lsi-helps-a-customer-move-operations-to-mexico/ ) has moved 4 air conditioner production lines to Mexico and is in the process of helping the customer move 3 more lines. One line was completely new equipment, while three have been a combination of existing and new equipment.
  
• Booked a boiler and steam production optimization study at a Memphis process plant, which could lead to major costs savings in steam production for the manufacturer.
  

The thing that is most striking is that most of these efforts were executed by multi office teams that comprised of engineers from more than one LSI offices, and in many cases, 3 LSI offices. It is truly a case of “the whole is greater than the sum of its parts” because LSI is getting the right people on the right job at the right time, irrespective of which office that the people are from. It is that cohesiveness that makes LSI very unique in the system integration world.

Eliminating the White Knuckle Ride of Startup: Simulation and Factory Acceptance Testing, Part II

 

In part I of this blog, the importance of having a good functional description was essential in developing a good simulation application, which then leads a good factory acceptance test. In this blog, the importance of simulation is discussed, along with simulation techniques.

Simulation of the software application is a critical step in any project in LSI’s view. The goal of simulation is to eliminate as much of the unknowns as possible before the most expensive phase of the project begins: startup and commissioning. As one of our customers puts it: “the goal is to eliminate the white knuckle ride of startup.” The other goal is to provide the right level of simulation to have a good factory acceptance test (FAT) or software acceptance test (SAT). The same LSI customer from above offers this perspective: “Simulation allows us to exercise all anomaly conditions of our process without real world damage.” Their application involves running gas compressors on a large pipeline. Emergency shutdowns can be catastrophic if not handled properly. Proper simulation allows for these conditions to be analyzed and taken care of before the control system is applied to the real world application.

A good simulation accomplishes things like:

• Confirms that interlocks are working properly
• Tests the sequences in the process or machine to make sure the steps are executing properly and that the proper transitions between steps are identified
• Confirms that any faceplates or pop-ups work correctly on the human-machine interface
• Any bugs in the code are detected and fixed
• Confirms the alarming strategy

Simulation techniques vary rather widely and the proper technique needs to be applied depending on project complexity and the project needs. As a user, you truly want maximum gain with minimum cost, so choosing the right technique is key. There are generally three types of simulation techniques applied and they are:

• Low Fidelity
• Medium Fidelity
• High Fidelity

In low fidelity simulations, simple “tiebacks” are used (i.e. when a valve is called to be opened, the open switch signal is triggered via a simple tie-back from the open output after a set time) and there is typically logic written in the control system to perform these “tiebacks.” Most of the time, these simulation routines in the PLC are removed after Factory Acceptance Testing. DCS systems often have a “simulation mode” built in. Many times it is up to the engineer doing the simulation to manually intervene to respond to process changes (i.e. if a pump that is filling a tank is called to run at a higher speed, then the tank level fills faster – there will have to be some interaction to make this happen). This technique is used for simple processes or machines where there is not a lot of loop tuning and loop interaction.

Medium fidelity simulation is much more process centric and takes things like mass balance, material balance, and heat balance into account. To continue with the example above, with the tank filling algorithm, a medium fidelity simulator would take into account that the pump is running faster and that the tank will consequently fill faster. This type of simulation is often used in process applications where flows, temperatures, and pressures need to be simulated, but the complex interactions that can take place between these variables does not need to be modeled. In medium fidelity simulation, it is not expected that loops will be tuned and loop tuning will need to be done at the time of commissioning.

High fidelity simulation is the most time consuming and cost intensive approach, but it can more than justify itself when doing a highly complex, integrated process. For instance, in simulating a power plant and its dynamic response to step changes in demand, it is critical to understand how the plant will react. The engineer and the operator would like to know what the impacts would be to steam flow, boiler drum levels, turbine speeds, etc. A high fidelity simulator literally simulates the entire plant and all of its interactions as it would happen in real time. The operator literally would see no difference in the simulated plant versus the real plant. Certain industries such as power and petrochemical require this level of simulation as it is critical to know how the processes will react together before the system goes live. It takes a lot of time and money to get the simulation completed in a high fidelity simulation. In industries like power and petrochemical, it is essential. In simple batch processes or machines, it is overkill.

The critical piece in all of this is to evaluate what makes sense for the application. LSI has used all of the techniques above. LSI writes a fair amount of simple tieback logic for simple batch processes and simple machine control. For medium fidelity simulation, LSI has used products like Mynah’s MiMiC (http://www.mynah.com/), Cape software (http://capesoftware.com/), and other products. For high fidelity simulations, LSI has used the above products, or has used more “home grown” systems from LSI’s partner company, Enero Solutions (http://enerosolutions.com/dynamic_simulation.php). (LSI and Enero partner together on steam optimization projects in power plants and cogeneration applications in plants like paper mills). During each project, it is imperative that LSI and the end customer evaluate the level of simulation required to streamline the startup and commissioning process.

More resources to understand simulation, how to apply simulation techniques, and what products are available can be found in the following locations:

http://www.mynah.com/content/understanding-simulation-fidelity

http://www.mynah.com/content/developing-medium-fidelity-dynamic-simulations

http://modelingandcontrol.com/

http://enerosolutions.com/dynamic_simulation.php

http://enerosolutions.com/simulation_steam.php

http://capesoftware.com/

http://www.mynah.com/

 

 

The importance of Simulation and Factory Acceptance Testing, Part I

I want to outline some critical steps in projects that many end users still overlook, and those are functional description definition, process simulation, and factory acceptance testing. I think that the value of a proper simulation and factory acceptance test are often overlooked by many end users when they are executing projects. However, over the years, LSI has found that proper simulation and factory acceptance testing is critical for many reasons within a project and they are not limited to the following:

• Reduced time on-site testing and troubleshooting software
• Faster start up
• Less end customer changes during startup
• Less stressful/smoother startup

In part I of this blog, we will discuss the first step in having a good simulation and factory acceptance test, and that is having a good process/functional definition. Certainly, the first component to have a good simulation and factory acceptance test is to agree on how the process works and what the end user wants to accomplish during the project. The key is to document the process and how the operator will interact with the process on the front end so that clear expectations are set and so that the software can be written as efficiently as possible. This is true of a new process or of one that is undergoing a major retrofit. I can think of two examples of where the process is existing, but where the process definition and functional specification process has been or will be critical to the success of the project. One of those is a process in a steel mill that has not changed for many years and is almost all discrete functions. The other is a feed mill that has become heavily batch oriented in which the process has changed drastically since the plant’s inception. In the first case, the code was written by the OEM in such a way that made it difficult to troubleshoot. As a matter of fact, this customer states regarding the importance of a good functional description: “Changes and improvements that have been performed on various operating systems (fluid, pneumatic, electrical, control, etc…) will impact how the original system function has changed.  In order to properly prepare for project development and implementation, you must ensure that all impact and aspects are included.  The functional description is a necessity in order to ensure all parties involved understand and agree how the system will operate.“

In the latter project, 7 different programmers have touched the code and have had 7 different ideas on how to accomplish things. There is code in the PLC that is simply “dead” and does nothing at all. In both cases, the end customer wants a readable, functional, and expandable set of code in the end. The processes are vastly different, the PLC brands are different, but what is constant is that both have agreed that defining what the process does and how the code will look is a critical step in the project. From the end user standpoint, they will gain the following benefits:

• They will be absolutely positive how each and every part of the process functions and how the software was written
• The maintenance and operations staffs will get exactly what they want because it is well defined
• They will have an operator training manual for training new maintenance and operations personnel

LSI has used many techniques to document processes, from flow charts, logic diagrams, HMI screen samples, and in some cases, just a plain description of the process with words. At certain times, our end users have a format that they would like us to follow, and in other cases, there is not a predefined format to follow. The most successful way to document a process and how the operator will interact with the process through the HMI is to be as graphical as possible. For that, logic diagrams and flow charts (we have even pasted sample Sequential Function Chart code from the proposed PLC program in the document as a flow chart) are extremely useful. The most important thing is that a customer can understand what is being conveyed very easily, and this is best done by producing something graphical in nature. Another benefit of this approach is given by Nick Riggio, LSI’s Golden, CO branch manager: “[taking the graphical approach above] allows for the customer and us to have code reviews along the way. They actually get to see what their code will look like in the end and comment on it during the project.” Nick also states the following: From a technical standpoint, we have to detail all of the interlocks, permissives, etc. in the process. We can also drive out any anomalies that affect the system as well. The customer will end up with more organized, robust, and easier to troubleshoot code.”

In part II of this blog, I will discuss simulation techniques and why simulation is important; and in part III I will discuss what an end customer can gain out of a good factory acceptance test.

First video blog

 LSI’s first video blog – as a test.

LSI’s services may qualify for a tax credit

Recently, our CFO sent an internal email request about hours spent on a few “products” that LSI produces, as these hours may qualify for a tax credit. He also informed me that our customers also could receive a tax credit for hours that LSI has spent on their projects. So, below, I am giving a very simple, maybe even over-simplified, overview on how this tax credit could apply to a manufacturing company.

The credit itself is called the Research and Experimentation Tax Credit and was enacted in 1981 and it allows companies that perform technological research to get a write off on their taxes on researcher’s wages, their supplies, and a portion of subcontractor labor (up to 65%), as long as the work is performed in the U.S. This tax credit was reinstated for the 14^th time since 1981 on December 17, 2010 as part of H.R. 4853, the “Tax relief, Unemployment Insurance Reauthorization and Job Creation Act.”

Basically, there are four criteria, which I will summarize below:

1. Permitted Purpose
2. Elimination of Uncertainty
3. Technical in Nature
4. Process of Experimentation

You can read a summary about each of these criteria at a CPA website here. Also, here is a case study about how improving a manufacturing process may qualify for this tax credit. What this means is that if your company has engaged in activity where research and development was done to improve a product or manufacturing process, then you may qualify for the tax credit. This could include the work completed by LSI or other outside contractors. The key is to have a well documented case, and to have accounting principles in place that allow these R&D costs to be tracked and documented easily. The burden of proof is on the tax payer, therefore, you should contact your tax professional to investigate if and how this tax credit can be applied to work in your facility or facilities within the last year.

Useful links on the subject matter (all were used in the information provided above) are below:

Article on the irs.gov website

Article on inc.com

Article from R&D Magazine

More Information #1

More Information #2

You can find more about LSI here. LSI is also on Facebook,
Twitter, and LinkedIn.