work servers hacked

Discussion in 'Bad Dog Cafe' started by GunsOfBrixton, Sep 15, 2021.

  1. Telekarster

    Telekarster Friend of Leo's Gold Supporter

    Posts:
    2,600
    Joined:
    Aug 14, 2019
    Location:
    Earth
    This is a great post man, and thanks for sharing!
     
    MrWiggly likes this.
  2. ghostchord

    ghostchord Tele-Holic

    Age:
    53
    Posts:
    649
    Joined:
    Aug 5, 2020
    Location:
    Vancouver, BC, Canada
    Sorry to hear. That sucks. We have a big issue in the industry around security. I think there are practices that effectively eliminate this risk, securing your network, securing your data, backups, training etc. etc. but those are not widely adopted.
     
  3. getbent

    getbent Telefied Silver Supporter

    Posts:
    44,642
    Joined:
    Mar 2, 2006
    Location:
    San Benito County, California
    That strategy won't work anymore. The challenge now is that the ransomware folks get in and just hang out and infect you for 6 months. By then, they are in any backup you might have.

    When you restore, they are signalled and MAYBE they lock you down again and maybe they don't. Usually, they will wait another 2-3 months and get you again. By then, you have invested but, you are still at risk, so they make you look terrible. At that point, you pay. and because they let you go the first time, they double or triple the cost.

    I do not believe that there are a set number of things that will protect you to the degree you feel safe. If there are humans in your system, you are vulnerable. The systems are really just for humans... so, it is brutal.

    The hope used to be to be just a step faster than your neighbor and he'd get eaten. Not so anymore. Now, you can rent and lease the software and there are so many concurrent attacks running all the time, it is much more random.

    The internet used to be like an airport in 1970. Fun! Now it is like an israeli airport today. Users don't notice that much of a difference but the BOFH's 100% feel it every single second of every single day.
     
    buster poser likes this.
  4. ChicknPickn

    ChicknPickn Tele-Afflicted Silver Supporter

    Posts:
    1,323
    Joined:
    Apr 16, 2007
    Location:
    Ole Virginny
    Indeed. The old Grandfather-Father-Son system was the standard for a long time, and one prayed malware wasn't baked into those backups. It's hard to imagine anyone being able to continue operations after restorating from a one- or two-year old backup, and I'd seriously hate to try to rebuild a business that way. People have asked me to consult in the years since I signed off. I've responded with a resounding no. So what, then, is the answer? If I knew, I guess I'd be very, very wealthy. But one thing I can say is that you didn't have this crap in the days of mainframe computing. Giving every employee internet access is absurd. It is doubtful even that most employees need email access to the outside world. A paradigm overhaul has to happen, and in the end, it may make greater use of closed, proprietary systems. The PC revolution has come to its inevitable nadir - - it's time to keep data, and employees, behind thick walls.
     
    Jerry_Mountains, getbent and Danb541 like this.
  5. GunsOfBrixton

    GunsOfBrixton Friend of Leo's

    Posts:
    2,086
    Joined:
    Dec 9, 2011
    Location:
    Rochester, NY
    It is going to be an interesting few days. The FBI are on their way (not that they will be of much help) and our sys admin is pulling the backups together. Right now it looks to only be the web servers and the DB server is still safe. The COO and IT head are still avoiding taking charge and pushing it down. The owner is just yelling (via email) at everyone. Typical day.
     
    getbent likes this.
  6. ChicknPickn

    ChicknPickn Tele-Afflicted Silver Supporter

    Posts:
    1,323
    Joined:
    Apr 16, 2007
    Location:
    Ole Virginny
    I hope that your web servers are like ours were - - not housing sensitive data. If we had doubts about web servers, we just spun up clean clones. The DB server is the real prize. You can celebrate if that one is safe.
     
    getbent and NICQ like this.
  7. MrWiggly

    MrWiggly TDPRI Member

    Age:
    71
    Posts:
    38
    Joined:
    Feb 12, 2021
    Location:
    Michigan
    This why I retired as well. I can truly say I never, ever miss the job. I still love tech and get a lot of satisfaction from helping out friends, family and acquaintances deal with tech issues. At one point in my career I single handedly built a wide area network operations and monitoring center for a networked that spanned form San Diego to Bath, ME and many branches in between. My job was to analyze traffic in simulations of equipment interoperation on a model of an on ship backbone for a new class of amphibious assault ships for the Navy. I'll set aside the latency discussion as the design was set in stone before I came on board to make it a reality. I sure don't miss the headaches I suffered on that project.

    The other part of day to day operations was simply monitoring traffic because every engineer on the project in over a dozen locations nationwide had to pull their drawings every morning and file them every evening on a single service located in Louisiana. Again, not my design for the network but I did have to educate top level management on the relationship of latency to bandwidth as affected by windowing between Sun workstations and Microsoft servers. The top guys wanted me to bend the laws of physics so they could get extremely low nationwide latency at zero investment. I was in a position where I had to repeatedly tell these stuffed shirts that what they wanted was impossible and at the budget they were working with it was laughable. They'd done a nationwide search for someone to fill this position and were contractually obligated to the Navy to fill it. I happened to be right in their backyard so they were ecstatic to find me. Unfortunately for them I was well aware of the situation before negotiating my contract. They were unprepared for my salary requirements to put it mildly. At one point when they rejected my demand I just said, "Well, OK. I'll stay for two weeks and then you're on your own." The project manager caved the next day.

    I was so happy to get out of there. I had no job lined up but just wanted to get out of the mad house. The next day my phone rang and the university I had previously worked at wanted to know if I was looking for a job by any chance. I got recruited as the tech consultant for the computer science department. Ah, the nice, peaceful work of academia for a few years and then early retirement. Glad to be away from the madness.
     
    Jerry_Mountains likes this.
  8. ChicknPickn

    ChicknPickn Tele-Afflicted Silver Supporter

    Posts:
    1,323
    Joined:
    Apr 16, 2007
    Location:
    Ole Virginny
    Glad you were able to have the soft landing. There needs to be a recovery program for former IT folk. When I hit the wall, I dreaded every day. As they say, when we’re on our deathbeds, it is not likely we’ll be saying, “if only I could have one more day at the office.”
     
    getbent and MrWiggly like this.
  9. ghostchord

    ghostchord Tele-Holic

    Age:
    53
    Posts:
    649
    Joined:
    Aug 5, 2020
    Location:
    Vancouver, BC, Canada
    The point is that your data in the backup is not encrypted. So you *can* recover that data from the backup even if the attacker has been in your system. The general pattern here is that the people who do not have the right practices in place are the ones being attacked, and there's lots of them. Most companies that actually have good security don't get into this situation. Security professionals know how this is done, the people who get hacked don't want to invest in that aspect...

    EDIT: (or are just completely ignorant...) and yes, the network is not the same place it used to be (if it ever was).

    More thoughts: Banks for example tend to have terrible security practices. Does the US even require PIN numbers on credit cards yet? The security of checks is laughable. Many banks do not offer/use 2-factor authentication. Often SMS is used even though it's well know that it's not secure. There's many other examples. The other problem for your average small business is that they can not rely on the security of the things they buy (e.g. equipment or software or network connected devices or cloud services) which makes it almost impossible for them to be secure...
     
    Last edited: Sep 15, 2021
  10. archetype

    archetype Fiend of Leo's Ad Free Member

    Posts:
    6,519
    Joined:
    Jun 4, 2005
    Location:
    Western NY
    I understand your meaning, but know that risk cannot be eliminated. Ever.
     
    getbent likes this.
  11. archetype

    archetype Fiend of Leo's Ad Free Member

    Posts:
    6,519
    Joined:
    Jun 4, 2005
    Location:
    Western NY
    Assume that none of the servers or workstations are clean. Rebuild everything.
     
    glenlivet and getbent like this.
  12. dhodgeh

    dhodgeh Tele-Meister Silver Supporter

    Age:
    66
    Posts:
    131
    Joined:
    Dec 1, 2019
    Location:
    Fla-USA
    Yep. Isolated VM Hypervisors can be your friend here.

    When I had worry about such things a couple of years ago (the whole cyber security thing was a major reason for me to retire), we contracted with a company whose service was to keep employees education about cyber security threats. The company assisted with targeted phising test in order to gauge effectiveness of the training.

    All new hires had to complete serious cyber security training before getting any hands on with our computers.


    D
     
    ghostchord and getbent like this.
  13. ChicknPickn

    ChicknPickn Tele-Afflicted Silver Supporter

    Posts:
    1,323
    Joined:
    Apr 16, 2007
    Location:
    Ole Virginny
    That's painting with a very broad brush. Banks in the US are federally regulated, and IT audits were becoming increasing sophisticated in my last years of service. Penetration testing could go on for weeks, and the requirements for documentation of security practices were crushing. So my opinion would not square with yours. Security work is never finished, for banks or anyone else.
     
  14. BigDaddyLH

    BigDaddyLH Tele Axpert Ad Free Member

    Posts:
    59,250
    Joined:
    Sep 1, 2009
    Location:
    Kelowna, BC, Canuckistan
    I'm not encouraging illegal activity, but who would you rather be? The hacker or the grunt with his finger in the dike?

    [​IMG] [​IMG]
     
  15. aging_rocker

    aging_rocker Friend of Leo's

    Posts:
    2,706
    Joined:
    May 8, 2019
    Location:
    The back of Bill's mothers'
    Sounds like a real bad day, @GunsOfBrixton :(

    As everyone says above, securing your data is getting harder by the day, but that doesn't mean we shouldn't try.

    But yeah, if you have people, you have vulnerabilities. That's how 99% of the nasty stuff gets in, someone just opens the door for it.

    Unfortunately, this case just highlights the fact that a lot of businesses (and plenty of cash-strapped government departments...) just won't/can't spend the required time and money to at least make a recovery possible, and the people writing the malware are a lot better paid than than the IT pros trying to keep them out.

    And yes, generally they do release the data if you pay them, other wise no-one would pay up. But they probably keep the keys....
     
  16. ghostchord

    ghostchord Tele-Holic

    Age:
    53
    Posts:
    649
    Joined:
    Aug 5, 2020
    Location:
    Vancouver, BC, Canada
    For sure. An asteroid can fall on your offices. A flood can submerge all your servers under water. A fire could destroy them. Life is about managing risks... You get insurance for the things that are important but low probability events...
     
  17. ghostchord

    ghostchord Tele-Holic

    Age:
    53
    Posts:
    649
    Joined:
    Aug 5, 2020
    Location:
    Vancouver, BC, Canada
    The regulations always lag reality. Meeting the regulations and having actual security is not the same thing. But sure, there's a lot of costs for banks to meet the regulatory requirements. But they have some incredible amounts of money stolen from them every year, they just treat that as part of the cost of doing business. I mean is there anyone here who doesn't know someone who had their credit card info stolen and used for some transactions? Raise your hand ;) This is not a force of nature, it's just something the banks choose not to address because it's cheaper for them to do so...

    EDIT: and sure, banks have better security than your random business... the other thing the big banks have going for them is that you'll betcha the authorities are gonna be going after whoever tries to ransom a big bank.


    "Merchants in the United States are losing approximately $190 billion a year to credit card fraud - much of it online, according to a 2009 Lexis Nexis study" - https://www.forbes.com/sites/haydns...ual-fraud-scam-more-on-jumio/?sh=7a326eb9390e (and that's 2011! actually 2009...)
     
    Last edited: Sep 15, 2021
  18. getbent

    getbent Telefied Silver Supporter

    Posts:
    44,642
    Joined:
    Mar 2, 2006
    Location:
    San Benito County, California

    what do the hacker and dike look like?
     
  19. GGardner

    GGardner Friend of Leo's

    Posts:
    2,997
    Joined:
    Jun 22, 2017
    Location:
    NJ
    I wondered if anyone ever thought of that! The idea of sending out a conspicuously fishy email and seeing how many employees opened it and how many, if any, reported it occurred to me as I was sitting through a mind-numbing employee security awareness training webinar.

    My second thought was that it would be clever for a hacker to gain access to a big company by posing as an entity that offers a mind-numbing employee security awareness training webinar.

    My third thought was that my second thought wasn't so original--it would be akin to the folks who service your elderly parents' home alarm coming back to rip them off.
     
    SRHmusic likes this.
  20. ghostchord

    ghostchord Tele-Holic

    Age:
    53
    Posts:
    649
    Joined:
    Aug 5, 2020
    Location:
    Vancouver, BC, Canada
    It's common practice. Companies do this all the time (fake phishing campaigns). You defend in layers. This is the outer layer. Then you want to make sure that even if a single employee is hacked the attacker can't get access to anything else. The assumption should be that some of your employees will click on links... In the olden days the default assumption was that your internal network was secure. Now the assumption is that the hackers are already on your internal network, just like they might be on the Internet, and that you should secure anything internal the same way you'd secure it if it was directly on the Internet...
     
    GGardner likes this.
IMPORTANT: Treat everyone here with respect, no matter how difficult!
No sex, drug, political, religion or hate discussion permitted here.