Don't ever work on a COBOL modernization project. It is a career killer with little to no upside. COBOL systems and the people directly using them are Chernobyl.
Both government regulators and operations personnel have made careers out of trusting black boxes that no one fully understands anymore.
Regulatory specifications are ambiguous and disorganized. Imagine working with someone who knows that you are trying to shed light on the technology that they were responsible for, yet do not fully understand. How cooperative do you think these people will be with your efforts? How do you think the managers responsible for this work survive the politics of the project's inevitable failure? Blame rolls downhill. Modernization projects require strong leadership and collaboration across many teams. That's a huge problem because such an environment very likely does not exist in the world today. No one understands how the algorithms work or even how they should work! Forget how challenging it is to work with COBOL, which someone could eventually figure out. Whatever you've written has to be tested, and you don't have a trusted source of business logic from which to verify how it ought to work. You only have the black box.
If you've been given this task, start interviewing at other companies. You were given a suicide mission by people who are well aware of that.
My experience with this was with regulatory margin trading systems supporting a multi-billion dollar market.
There’s a lot of that out there. I have avoided COBOL but I have seen stuff written in VB6/COM which was also in the same state. Eventually the one guy who knows anything about it leaves or dies. In the name of “progress” someone wraps it in something else and sticks a web API around it but everyone is afraid to touch that bit of VB code. Now a third party calls this and wonders why it takes 10 seconds for an HTTP API call to run and they can’t be issued in parallel or it throws a 500 error. Then the third party asks to send 20,000 of these a day and then finger pointing and stuff occurs at management level between both companies.
I have been on both sides of this and it sucks. I came up with Rug Driven Development as a title. All the risks and future problems are swept under the rug quickly while maintaining a happy smile and pretending everything is just fine and dandy.
I so need to steal this term. Not exactly for what you are describing even if I come across this on a nearly weekly basis in so many different industries. Be it automotive, be it finance, be it what ever.
But - and this is were I will probably use it in the future - I come across it in agency work. You win a shiny new project proposal and need to quickly show something for your corporate "partners" to shine before their management. To make them look good for their yearly appraisals or what not.
You already know that you will never fix the underlying shit you built, because your contract will be a new pitch three years down the line and it isn't clear if you will win it again. So let the next people deal with the tech debt - but guess what: They will just do the same. So debt will pile on and on.
So "Rug Driven Development" will be my goto term going forward.
Rug Driven Development causes the majority of technical debt that I see in most of my clients, and it happens at the small scale, too. Just a couple weeks ago, a junior engineer was faced with a key process that segfaulted about once a minute. No worries though, they reasoned; the process immediately respawned and because it managed a stateless procedure, nothing is lost. Except the 1 out ~30,000 times it didn't respawn. No worries though, they reasoned; they put in a request to the monitoring team to look for the absence of the process for longer than three minutes and auto-spawn it. Ta da! Problem solved!
They let this go on for about six years before I arrived and saw it. It took me pointing out that there is a decent chance this is caused by the process relying upon something in the OS kernel that isn't quite aligned to the kernel documentation but is good enough, and the immediate respawn behavior might rely upon that in turn. If the kernel ever is "fixed", then this audit-compliance-related process suddenly stops and doesn't respawn, or worse, won't start at all, becoming top of mind with all the senior management. It is always cheaper to fix problems in the small before they become problems no one can ignore.
The urge to sweep problems under the rug and move on is very powerful within our industry. Until you've done the same yourself and been bitten so many times, your scar tissue twitches every time you see the same pattern again. These days, I treat code problems like I treat cleaning up messes while cooking: I clean up as I go along. My scar tissue thanks me now. There is a delicate balance between addressing these problems in the small and bikeshedding, though.
BTW, I performed an strace that revealed a SIGKILL just pops in without anyone or any known process issuing it. The application developers suspect something in the OS, so we're now engaged with the application support team, OS support team, and our own internal OS support team to trace it down and beat it into submission.
> I have avoided COBOL but I have seen stuff written in VB6/COM which was also in the same state. Eventually the one guy who knows anything about it leaves or dies. In the name of “progress” someone wraps it in something else and sticks a web API around it but everyone is afraid to touch that bit of VB code.
I call such things "Ancient Wonders". As in artifacts from long ago that the company owns, but nobody knows how they work or how they were built, and nobody can build one today. There may, in fact, be only one left. Whoever did build them was privy to some Tribal Knowledge that has since been lost.
I worked at a shop whose Ancient Wonder was their custom templating engine with an embedded Python interpreter. The only guy who knew how to compile it had left years before I joined, and as a result everybody just copied the same Solaris .so to new development/test/prod servers as they came in.
Read this story, "Institutional Memory and Reverse Smuggling" [0] - once upon a time, a petrochemical factory was built. Decades later, the plant is still operating and being maintained. But at this point, nobody knows how the whole factory worked, why it was built that way, which processes it ran or how it was constructed...
I have just straight up cleaned out my desk and turned in my notice once upon discovering something similar when delving deeply into the innards of an old embedded hardware stack.
It sounds like the majority of that rate is for knowing how to navigate an organization still using cobol. A fortune 50 using such a system as its backbone may be less cost sensitive than a FANG with only a small number of engineers responsible for holding the whole thing up.
Both government regulators and operations personnel have made careers out of trusting black boxes that no one fully understands anymore. Regulatory specifications are ambiguous and disorganized. Imagine working with someone who knows that you are trying to shed light on the technology that they were responsible for, yet do not fully understand. How cooperative do you think these people will be with your efforts? How do you think the managers responsible for this work survive the politics of the project's inevitable failure? Blame rolls downhill. Modernization projects require strong leadership and collaboration across many teams. That's a huge problem because such an environment very likely does not exist in the world today. No one understands how the algorithms work or even how they should work! Forget how challenging it is to work with COBOL, which someone could eventually figure out. Whatever you've written has to be tested, and you don't have a trusted source of business logic from which to verify how it ought to work. You only have the black box.
If you've been given this task, start interviewing at other companies. You were given a suicide mission by people who are well aware of that.
My experience with this was with regulatory margin trading systems supporting a multi-billion dollar market.