Real life experience shared by Joseph (Joe) Sebastian, our Principal Engineer / Regional Manager Eastern Automation and Controls about some of the issues Optimation has run into when diving into an engineering project, specifically discussing the importance of documentation.
So, you have a new system, and it is working perfectly. Did you get everything you need such as instruction manuals, maintenance schedules, spare parts, and how about accurate engineering documentation? When I say engineering documentation I mean things like schematic diagrams, bills of materials, risk assessments, and other documents that you probably don’t need for a while but will eventually. Hopefully you don’t need some of them ever, but chances are that at some point down the road you are going to need a piece of documentation that may have never been delivered.
It took me 30 years to figure this out, but I knew at some point a problem had to appear. Engineering in the 1990s was much different. I was new to the whole engineering game. I was young and I was ambitious. I would walk through the engineering department of very large manufacturers and see about a third of the engineers reading the newspaper in the morning. At the time I just thought they were lazy, but looking back I am assuming that all their projects were in good shape, completed, or a lull before the next downtime crunch, etc. One global manufacturer I worked at had hundreds of engineers in their corporate engineering department. Over the next 10 or 15 years I watched as those numbers consistently shrunk. At first it makes sense, get rid of the dead weight, or at least only keep the number of engineers you need to get the work done. Obviously if your engineers have enough time to read the newspaper you have too many engineers. The bad thing is that the reduction of their engineering department didn’t stop with just the dead weight. I assume after seeing the instant boost to their bottom line the company continued cutting.
At that time engineering companies like Optimation were blossoming. We were a consulting engineering firm and we filled in the gaps that were being created. This was working out nicely. Essentially, the client only paid for the engineering they needed. Their engineers were busy writing project specifications and requirements, while we were doing the detailed engineering and design. Unfortunately, the engineering cuts kept happening. Then with very few, or no engineers, consultants like Optimation were left to do it all. This is something we could do, but the interface with the client was difficult as there was no one left that knew the ins and outs of the client’s processes, or reasons they did things the way they had, or even how to run a large capitol project. This strained the process, but again the clients were willing to pay a fair price as it was still less of a hit to their engineering budget than having too many engineers on staff. A plus to this was that the engineering cost was more visible now and could be associated with the capitol project and not just an expense, so there were accounting advantages at tax time.
Now, the cuts didn’t stop at letting people go, the project engineering and design became treated as a commodity just like a nut and bolt. Companies started issuing project engineering to the lowest bidder without being able to specify wat was needed or being able to verify what they needed was actually delivered. I understand that no one wants to get taken advantage of, but do you hire the cheapest lawyer or the cheapest doctor? Anyway, many companies seemed to make this work, and I was living this as a consulting engineer. I watched as many competitors under bid us and got repeat work. Sometimes in the bid process competitors would even tell the potential client that if Optimation got the work that we would do a great job, but we still didn’t get the project. I started to see why; many clients didn’t even realize what they would need years after the project was complete.
Things like detailed accurate drawings is a simple example. If clients got drawings at all, they were the bare minimum to complete the project. Many decisions were made that were not “forward-thinking”. One example of such a decision is: “Just markup the drawings with the changes and we will have the drawing set updated later”. Unfortunately, later came and went and the hardcopy with the markups never got incorporated into the official drawing set. Another example would be “Renumbering all these wires in the field will be too difficult, cost too much, or will require too much down time so let’s make a cross reference so we can look up the field wire number and find it on the drawings”. The marked-up hardcopy of the drawings, or the cross-reference list, got lost, destroyed, or not updated when the next projects happened.
Fast forward to less than a year ago. I’m doing a large automation project for an international manufacturer and all of this hit like a ton of bricks. The system was large and was initially installed over thirty years ago. With all the different documents they provided, schematics, plant layouts, cable schedules, cable routing drawings, miscellaneous spreadsheets of information, and the automations system programming, we struggled. We had many hours in meetings as an internal team and could not get any of the different pieces of information to correlate. We took a piece of data and would try to follow it from one document to another and would invariably get lost so we would try with a different piece of data. It didn’t matter if we started with a valve, a wire, or a PLC input, we ended up totally lost. None of the documents lined up. This alone crushed the cost of the project, and the situation almost led to utter failure of the project as we had a fast-approaching deadline. My team could not even begin to do our work because we had nothing accurate to start with. We ended up having six weeks of morning meetings with the client. These meetings averaged an hour, some slightly less and some half the day and these were every single day with multiple people from both teams. We went through every wiring connection to the automation system and verified what still existed, what was added, and where each wire was actually landed. We ended up with a new spreadsheet outlining each node and every input and output of the entire system. I now refer to this document as the $150,000 spreadsheet.
When up to my eyebrows in this project, in the back of my head I kept trying to figure out what the heck happened to this system. How did it get so bad, how did this large, Fortune 100 company that knows engineering, end up with their engineering and design documentation in such bad shape? Well as the project evolved, I found out many things.
Mainly, the facility was not always one of their plants. It was purchased many years earlier and was known to not be on-par with the client’s usual level of documentation. After the purchase the new plant was left to its own poor practices. The main one was either no, or bare minimum engineering, on project after project, year after year, which added up to hundreds of thousands of dollars of engineering to remedy, just so we could start the project we were brought in to do. I finally saw the fallout from all these engineering cuts 20-25 years ago, and bare minimum engineering ever since. It was killing this project. Everything was developed directly from our $150,000 spreadsheet. It was imported into the automation system, used as comments in the code, used for alarm messages, creating tag names in multiple systems, and used to update and generate new schematic drawings. When we went online with the new automation system, there were only 2 errors in the thousands of points. We had missed 2 handshake signals between our control system and one of their process lines that had been added at some point over 30 years. The entire process yielded success but added a great deal to the cost of the project, and we were at risk of not meeting the corporate imposed deadline.
The on-site technical manager was brought in a couple years earlier to get the plant into shape as far as the systems and related documentation goes. I had a history with this person, and he targeted me and Optimation for this project because he knew we would not allow the engineering documentation to remain in its current condition. We couldn’t, with all that had to happen in a super short amount of downtime, this project had to come out of the gate running. You cannot do that without at least almost perfect documentation.
None of these situations were obvious at bid time, although no one seemed surprised. I have seen my share of crappy documentation, but never on a scale this large. I do have a gut feeling that we will run into this more and more over the years, and there are a few solutions. First, the client must know what is required to maintain a complex system, and demand that that is part of the deliverables on every project. They must also know how to verify they have received these items and be able to store and retrieve these documents as needed for future project. Hiring a consulting engineer to manage the project and ensure you get the required documentation from your contractors is a great option. The route that my client is using is to establish a relationship with us and have us be an integral part of their team to work through these systems and ensure through the upgrading of the systems that upon completion they have correct engineering and design documents. I am looking forward to working with this client, but the beginning of these projects is going to be rough.
Want to discuss more on how to ensure proper documentation of your engineering project? Reach out to Joe here: joe.sebastian@optimation.us
OR
{{cta(’81d8e030-97be-4420-ab03-a44363808a26′)}}