I'm in the mud
"Houston, we've had a problem"
I really love the movie Apollo 13, from Ron Howard. It was literally taught in one of my grad school classes (though not a very good one, but that's not the movie's fault) as an example of crisis management. The score is incredible, and I loved every one of the practical effects.
YouTube likes to show me scenes from it from time to time, and one of my favorites is the famous "we've had a problem". In that scene, Fred Haise (played by Bill Paxton) says "Houston, you're in the mud", meaning that he can't make out what they're saying.
Projects
I have a phase I go through with any project where I'm in the mud. I'm in the mud right now on this data lake project—I have interactive Spark sessions working well, some basic Delta Lake ETL happening, and gnarly typing stuff sorted out, but I'm still only about 80% done. The remainder of the ETL involves tuning both Spark and Delta Lake, though, which means optimizing the partitioning and worker characteristics to be able to process everything without timing out, costing a fortune, or crashing. And it turns out I might not even need to do it! (But that's a story for another day.)
It's a classic "the last 20% of the work takes 80% of the effort" scenario. Add ADHD and Long COVID brain fog, and you can imagine it's a trial.
When I'm in the mud, it's hard to communicate with myself, let alone anyone else. It gets hard to hear myself think, and I have a tough time communicating anything to anyone else. I find it's a common part of my process... but does it have to be?
Finding a path out
The best way to get out of the mud usually involves involves analog tools. As much I enjoy analog processes, I'm not dogmatic about them. When it comes to finding a way out of the mud, though, they're hard to beat.
I picked up a desktop whiteboard at the end of my workday to quickly diagram the logic for a conditional. Not a tough task under good conditions, but easy to mess up at the end of a long day. Lately, at the behest of the powers that be, I've been trying to use LLMs for that, with minimal success.
With the whiteboard in hand, I was done in seconds, with greater clarity. It reminded me that the process of writing and drawing by hand really clears out the cobwebs and helps me understand my own thought process.
So tomorrow, when I get back to my desk, I'm sitting down with an open notebook and a pen or pencil, and writing through my day.
Some more reading: