Here's the thing with data engineering, if you do an absolutely superb job you end up being largely invisible to the business. Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected.
The issues I encountered working as a data engineer many moons ago, is that in a business that's either agnostic to data quality or doesn't invest in reliability, the data engineers tend to become the face of failure. If a Kafka queue begins acting up, or a data ingestion jobs fails because of some upstream data source changes, then it's usually up to the data engineer to...
1. Notice that something's gone wrong.
2. Figure out who in the organization is going to be affected and let them know.
3. Run any triage and comms with the business keeping them in the loop until it's fixed.
In my experience, like the guy who only gives good news to the dictator, certain execs/leadership would quite happily let data engineers take the lead for outages, only swooping back in once everything was up and running again to demand a report on "how this happened" and some glib advice to "take time to stop this happening again".
While it can be technically rewarding as a career move, it's got to be highlighted that it's only fun and games until you hit your first major outage.
This is a lesson I keep repeating to our juniors: when you do something notable, celebrate it, and make sure to include the whole team, the users and the boss in the celebration.
It can be as simple as an email, but sometimes, making a few chat messages or even calls. It takes times, it seems silly, but you get a fantastic ROI on this.
Because now people see your work as valuable. It's concrete to them. They understand its value. And this has many positive consequences on your life.
This, of course, works only if the celebration avoids technical terms, is short, straight to the point, and explain what's the benefit to them and the org and how much it took to to it.
People must know that you are doing a good job, and you are the only one that is in the position to let them know. They will not get out of their way to learn about this.
E.G of a chat message I would send to my client's team boss:
Hello <first name>, it took 3 weeks, but we finally finished improving <system x>. This should save us <y> of <z> in the future. We tried hard to make sure your work is not disturbed by the change so it should be transparent for you. But let me know if you notice anything out of the ordinary. Cheers
Believe it or not, people actually like it. It keeps them in the loop, give them context and perspective, and help them take better decisions. So don't hesitate to do it.
> It keeps them in the loop, give them context and perspective, and help them take better decisions
Other side of the coin:
"I just spent 2 weeks optimising a daily process to be twice as fast" might well end up in me wondering why anyone would care about the 1 minute reduction in delivery time for daily cadence data. It might also result in me being stroppy about work I cannot get resourced that has obvious $ value.
But that is also a win for the business. Either I am wrong and am persuaded of the value, or I am not and I become more engaged with stakeholder prioritisation and we set up something more effective.
That's good feedback to have. If your boss/client step in and says "we don't need that", this allows you to re-calibrate the team priorities and shows you need to get better at extracting the real need in the future.
I completely agree with your initial comment and your response here. Transparency is a great feedback loop. And it's always good to celebrate what you do.
1) Break something on purpose. 2) Make sure the business notice it (usually a spectacular cost increase does the job).
3) Come in and fix the issue
4) Make a cost comparison quick chart before/after
5) Become famous
If you look long enough in most code bases, you will eventually find a spectacular problem.
Case in point, I was working for several months for a client, and I had a slow afternoon, so I decided to convert some calculation to numpy, see if we gained any free perf.
We got a x100 local speed up, which was very fishy. Gaining speed is common with vectorization, but two orders of magnitude is a lot.
So I looked at the original algo. There was a glaring mistake in it, that I fixed by my numpy code without noticing, and just changing that in the pure Python algo, without using numpy at all, made it X50 faster.
I could then call my client, and celebrate the good news. Not "there was a mistake", no. But "we found margin for progress".
This hits home. I communicate a lot more with up and downstream stakeholders as a data engineer. Be it for designing future changes and how those affect others. Or, as you said, triaging outages/delays, figuring out where the missing dependency is. Less problem solving by coding.
The SQL thing the article mentions is also true. Even if you work with higher level tech like Spark. SQL gives you a good feel for joins. I also like the fast feedback cycles you get from SQL compared to something Scala/Python based.
This is exactly why I found DE is not a long term career path.
I work in data, I occasionally do DE roles, but if you aren't moving over to the app side or business/domain side, you are leaving a lot of money on the table long term.
The funny thing, being in data, if you take the time to understand it.. you are tremendously valuable to the business/domain side.
Otherwise you 1000% risk becoming part of the background infra.
No one celebrates when the faucet dispenses water or the power switch works.
They do lose their collective #$#@ when the water or power fails though.
Data Engineering has a tendency to be treated that way.
DQ is a big part of it as well. Everyone wants it, no one wants to pay the hours/dollars to do it.. No one is sure what they want, but probably what you propose is too simple. Can't AI do it? Etc.
TBF, in most companies that have a traditional core business (let's say, selling pharma products), IT is a cost center and your run of the mill engineer is seen as the plumber who's suppose to make everything seamless, as you describe.
Short of solving a pain point in a spectacular way, the whole IT dept will be the face of stuff crashing, security issues and budget burning.
> if you do an absolutely superb job you end up being largely invisible to the business
openly communicate how difficult some feature, project or solution is going to be. estimate time then multiply with pi and round up. then proudly present how you got it done despite obstacles possibly even in less time.
The Futurama quote is relevant: "When you do things well, nobody will know you did anything at all". I think this goes for a lot of engineering, though. Especially infrastructure.
> Like the very necessary guys who fix the water mains or the sewers, the expectation is that the service/data pipeline/database will be always available, and nobody really cares too much either way once everything's working as expected.
The issues I encountered working as a data engineer many moons ago, is that in a business that's either agnostic to data quality or doesn't invest in reliability, the data engineers tend to become the face of failure. If a Kafka queue begins acting up, or a data ingestion jobs fails because of some upstream data source changes, then it's usually up to the data engineer to...
1. Notice that something's gone wrong.
2. Figure out who in the organization is going to be affected and let them know.
3. Run any triage and comms with the business keeping them in the loop until it's fixed.
In my experience, like the guy who only gives good news to the dictator, certain execs/leadership would quite happily let data engineers take the lead for outages, only swooping back in once everything was up and running again to demand a report on "how this happened" and some glib advice to "take time to stop this happening again".
While it can be technically rewarding as a career move, it's got to be highlighted that it's only fun and games until you hit your first major outage.