Huh?

  • 0 Posts
  • 27 Comments
Joined 10 months ago
cake
Cake day: June 8th, 2025

help-circle



  • I’m a data engineer, ofc I know that. But you were excusing it as if their service has costs. That’s bs, because their reported earnings that done without whatever profit they are going to gain from this harvest of train data already were very profitable.

    That’s why I said you were in the wrong. Not because you expected them to sell everything they could, so did I, but because you justified that behaviour from the free to play model. That model exists in plenty games that are extremely successful without harvesting things beyond metadata.










  • You don’t get hacking protection from bots, you get protection from DDoS attacks. Yeah some customers would have gone down, instead everyone went down… I said that instead of crashing the system they should have something that takes an intentional decision and informs properly about what’s happening. That decision might have been to clo

    You can keep the policy and inform everyone much better about what’s happening. Half a day is a wild amount of downtime if it were properly managed.

    Yes, bot detection is not the most critical…

    So you agree that if this were controlled instead of open crahsing everything them being able to make an informed decision and opening or closing things, with the suggestion of opening in the case of not detection is the correct approach. What’s the point of your complaint if you do agree? C’mon.


  • They probably mean that they did a change in a config file that is uploaded in their weekly or bi-weekly change window, and that that file was malformed for whichever reason that made the process that reads it crash. The main process depends on said process, and all the chain failed.

    Things to improve:

    • make the pipeline more resilient, if you have a “bot detection module” that expects a file,and that file is malformed, it shouldn’t crash the whole thing: if the bot detection module crahses, control it, fire an alert but accept the request until fixed.
    • Have a control of updated files to ensure that nothing outside of expected values and form is uploaded: this file does not comply with the expected format, upload fails and prod environment doesn’t crash.
    • Have proper validation of updated config files to ensure that if something is amiss, nothing crashes and the program makes a controlled decision: if file is wrong, instead of crashing the module return an informed value and let the main program decide if keep going or not.

    I’m sure they have several of these and sometimes shit happens, but for something as critical as CloudFlare to not have automated integration tests in a testing environment before anything touches prod is pretty bad.



  • *data engineer

    I’m also one but I don’t work for advertising. Most data engineers work for consulting companies that work for banks. We program automatic data processing pipelines. For example, bank transactions are stored somewhere, all the historic data, that needs processing to then be graphed out for exec number 3, or for whatever.

    Other companies might send you files that need to be automatically processed, cleaned, and put correctly where then other tools can pull that data correctly.

    We basically do all the background work concerning data manipulation. File processing, databases… all that stuff. And by databases it can be normal ones like posture to distributed ones like hdfs/hive/athena/whatever.

    Ad world is basically the same but with tracking info instead of transactions.

    If you are interested in day to day work, it’s a mix of coding SQL processes, then porting them to spark/pyspark for distributed massive processing. There are new shiny tools for those that don’t know much of the technical side to manage, sorta.