Automation Upgrades – Why are they so important?
Do you bring your trash with you to your new house?
Do you fix things that aren’t broken?
Knowing these answers can help get through your automation upgrade.
Optimation has become part of Re:Build Manufacturing and their Total Production Solutions group. The mission of our companies joining forces is to rebuild America’s manufacturing base, develop and protect valuable intellectual property, and create meaningful, sustainable jobs. To do this, American manufacturing must invest in their employees and their facilities. For us, that means upgrading their electrical and controls systems. At the heart of these systems there is a brain and employees who program those brains to run gobs of different machines, systems, or processes. It’s outstanding what we have pulled off over the years with these devices. In less technical forums we refer to these brains as controllers, while in the biz, we call them Programmable Logic Controllers or PLCs. These brains are connected to a bunch of sensors that monitor positions, temperatures, or other data and are the controller’s eyes and ears. The controller is also wired to a bunch of devices like motors, valves, positioners, and pumps that would be like your brain controlling your muscles. Hundreds of times a second the controller runs the program, looks at all the sensors and decides what to do with all the devices. Really cool, right? These PLCs are awesome and can run for decades in grueling situations, but they can fail and there is a limited supply.
Probably within 3 feet of you, is your phone. Look at what that can do. Look at what it can do this year that it couldn’t do 5 or 10 years ago. Your phone gets old and after a while you don’t get any updates, can’t get a new battery, a charger, or a replacement screen. These controllers or PLCs are the same way. Manufacturers tell you up front how long they will provide parts for your PLC, just like they do with your phone. Samsung supports their phones for four years and Apple for around seven. PLC manufacturers support their controllers for about 15 years. Just like your phone, once support ends you need to upgrade to a new one or risk going without. Many industries have gone way past the support period and are passing the point that they cannot get used parts from repair houses or eBay anymore. Now if this controller fails their entire manufacturing line may be down. To prevent this they need to upgrade their controllers every 15 years or so. Optimation does this quite often for many clients.
One of the projects we are currently working on is a program conversion from an old Modicon PLC originally installed in 1994. It seems that lately many of the projects we are doing include upgrading very outdated controllers to new ones. This is to mitigate the risk of not being able to replace the old, discontinued, controller should it decide to fail after 20 years. This usually requires programming for the new controller to function. One of the biggest and most frequent mistakes we have seen is the approach taken by clients when embarking on this journey. Typically, clients think that a new controller will just run the old program, or that this will be a simple program conversion. Sometimes this is an option, and it means that every rung, network, or line of code in the old controller, will be converted to a new one in the new controller. This is what we call a one-to-one conversion. Now, it is logical to think that this makes sense as the system has been running for many years and works just like they want it to so why re-invent the wheel? Let’s just run the program through a conversion utility, it will work the way it always did, and we will be done. Even if there is no automatic conversion utility, we will have someone go through the program bit by bit and convert it into the new program. This approach is fueled by the assumption that programming is just typing, and, well, that is a topic for another day. Anyway, reinventing the wheel is not really what we want to do, but there are many reasons to not do these one-to-one conversions.
The first and obvious reason to generate completely new programming is that sometimes a simple conversion is just not possible. Going from one brand controller to another or one programming language to another is not a simple conversion. There is no automatic conversion utility to do it for you and a human will need to be well versed in the old programming language, the new programming language, AND actually know what this system is supposed to do. This doesn’t sound difficult, but honestly, the last part is the toughest.
If it is possible, more times than not, it just isn’t worth it to use that old program. We know that sounds crazy, and expensive, but keeping the old code means that you are keeping all the trash that’s buried in it, stuff you can’t see from the outside. Like renting a dumpster before you move, only bring what is valuable to you to your nice new home. Many people don’t understand why the quality of the code is important. Our analogy is that everyone knows that their dishwasher has a microcontroller in it that runs a program (most dishwashers anyway). You don’t care what the programming looks like, as long as you get clean dishes. When you don’t get clean dishes, you call the repairman, and they fix the dishwasher. If there was a problem with the dishwasher program the repairman put in a whole new controller complete with a new program right from the factory. You just had to hand wash dishes for a week until the new controller arrived. Now say you have a business that has a big, custom dishwasher that runs 8 hours a day. Each week the payments on the dishwasher are $10,000, but it makes $56,000. It stops working and the repairman comes and looks at it. This is a custom and complex dishwasher, so they need to look at the programming to figure out why it is not working. The repairman charges $200/hour, the dishwasher payment is $200/hour, plus you are not making the normal $1000 per hour. With a $1400 per hour cut to your cash flow, how long do you want the repairman to spend digging through the code? Not long, we bet. Concise code allows the repairman to troubleshoot and get you back to making money quickly. Over the course of the dishwasher’s life, it will stop, many times. It doesn’t stop because of the programming (usually), it stops because of stuck valves, plugged water lines, a busted door switch or other issues that are all easy to diagnose with a properly engineered program.
We have seen so many examples of poor programming that I can’t even remember them all. Redundant, or inefficient code is bad, but probably the worst is dead code. The program we are working on now had six networks of calculations just to prevent overcurrent on one fan. They looked like this:
If you have no clue what these do then you understand my point. Even if you know this type of programming, you know that it will take some time and research to figure out exactly what is going on. After spending hours digging through the six networks and all the other ones connected, and trying to make sense of what they do, we concluded that they don’t do anything. The client confirmed that this functionality was attempted and never commissioned because they couldn’t get it to work. What they were trying to accomplish was something we had done many times before, we did it in one rung of code in the new controller.
Easy to see what’s happening there. It’s just math. If you don’t understand that either, that’s ok. It does look better though right? A hint for you is to look at the title highlighted in cyan. The first three words give it away. The point here is that instead of spending hours chasing through the program, it is clear and concise what is happening. This saves time, and in the case of the complex dishwasher, that time is worth $1400 per hour. In many places that we work, this would be a tiny fraction of the real cost.
Another reason to trash the old code is that old code has old instructions. The example above shows this argument too. We used one instruction to do the math. If that had been converted automatically or in a one-to-one conversion it would have taken the same crazy number of instructions to do the same thing. Just figuring out the data flow from one instruction to another is difficult let alone what the instruction is actually doing.
More trash that is carried over in a one-to-one conversion is undocumented code. After decades of a program running, a lot of code was probably added on the fly and someone intended to go back another day and document what it was doing. Unfortunately, that day never came so you can’t tell what the code is doing, what they are trying to do, or even if it does what they intended. It actually looks like they never even finished the code. (Yes this is really out there running right now)
If you can’t make sense of this, you are in good company. Some people out there can get the gist of what someone was trying to do, but there is so much wrong with this we ultimately just abandoned it.
Another reason for code trash is the physical equipment that does not work, or never worked, and uses code that may take many hours to decipher. So, it’s like dead code, but it’s dead equipment! Years ago, on a project, during a high-pressure startup, we couldn’t get a valve to operate after one of these upgrades. We called out maintenance and when they came, we met with an operator that had been there for decades. After repeatedly explaining which valve we were referring to, the operator proceeded to tell us that, although it is still on the roof, piped into the system, and wired into the controller, it is probably rusted in position because they haven’t used it in 15 years and will never use it again. Good thing they spent the money to have us upgrade that.
After going through programming others have written many times, while looking at it, you try to get your thought process aligned with the person that wrote it. This even holds true for documents like say a 1900’s novel, or the US Constitution, or the Bible. You may even do this without realizing it. You try to get used to their way of thinking, the specifications or limitations that influenced them, and in the case of this type of programming, how they implemented basic things like starting a motor, or processing an alarm. After a system has been running for 25 years, take a guess as to how many changes have been made to the programming. Then think about how many engineers, programmers, technicians, electricians have made those changes. You cannot get into a specific mind-set when hundreds of changes have been made by dozens of different people over 25 years. You can’t just look at the code and say “Oh look, that is just like what they did over here.”
“Reverse engineering this code is like taking dog vomit and trying to figure out your four-course meal that it had for dinner.”
All of these situations add up to code you probably do not want to bring along to your new controller. Trash it! You won’t even need to rent a dumpster! We learned this the hard way when we worked on an extruder upgrade over 10 years ago. Two of our senior engineers spent three weeks digging and documenting the code in an attempt to convert it. The code was terrible. It was so difficult and in contradiction with the client’s appraisal. We decided to trash it and completely develop new code from the ground up. To this day we talk about that, and we wish we had made that decision sooner.
But there is an automatic convert utility! We just click the button and it is done right? These utilities are worse than a one-to-one conversion by a human! They will convert about 90% depending on the original programming. Then you need someone to double check the code, run through every error and piece of programming and be sure it all is converted properly. This means you are paying someone to figure out what the original code is supposed to do and verify the new code is doing it. If you are going to pay for all that, just have them program it properly so you don’t bring all the trash along. Remember, the conversion utility is designed to yield a program that the new controller can understand, not necessarily a program that a human can understand. The end result is a program that is even more difficult to understand and follow (from a human point of view) than the mess of a program you started with. We are involved in a project that has suffered this type of a conversion at least once with a program that has its roots in the early 1980s on a DEC VAX microcomputer. It was then reprogrammed in an Allen Bradley PLC2. Then at some point run through a conversion utility to an Allen Bradley Control Logix PLC. As you can imagine the code was a disaster. Forty years of all the same issues described above. The company had hired a new technology advancement manager and he knew this had to be remedied. The actual PLC was a brand new model with a 15 year lifespan ahead of it, but he knew the programming was not worth keeping. They could not troubleshoot problems, or even add or change any functionality. They couldn’t add quality control, they couldn’t add new products, and they couldn’t change existing products. Their process was completely limited by the poor condition of the programming. We are finally on the tail end of the process and have converted about 90% of it to current Allen Bradley PAX objects. These are blobs of code that have built in functionality to run equipment and provide that to an operator on a display with an amazingly small amount of programming. It is incredible how easy the code is to troubleshoot and modify. The functionality for the operators and maintenance crew really streamlines troubleshooting and system modifications. This project literally advanced their manufacturing 40 years.
Remember, it is what’s on the inside that counts. This is true for programming too. If someone is thinking the code works now, why fix something that’s not broken, it is broken. Your car will run without oil. The chair is just fine with three legs. Your house is ok with no paint, siding, roof, etc. Do you fix these things? You should because they will end up costing more money in the long run. It’s the same when it comes to programming controllers. A great deal has changed in just the last 10 years. More programming languages, organizational structures and tools are at your disposal. They are tried and true, tested and understood. Using them in the code provides much quicker code development (Optimation) and much faster troubleshooting (repairman, probably also Optimation). Using a single function call over and over in the code means that you only have to troubleshoot that function one time and it will work every time for every device. Once a technician knows how that motor is programmed, they know how every motor is programmed because they all use the exact same function. Now they don’t have to dig through the code that runs the motor, they just look at the data for the motor and can instantly tell why it isn’t running. Oh, and to program a motor, just call that function again and point it to your new motor. You do all of this to ensure your manufacturing continues to run, it experiences minimal down time, you save money and become more competitive which means you ultimately make more money.
Just like moving to a new house. When you upgrade your automation system, don’t bring the trash with you.
Contact Us Today to help with your Automation Upgrade!
Form located in the contact us page