https://www.bbc.com/news/articles/cv2g5lvwkl2o
QuoteMass IT outage affects airlines, media and banks
A raft of global institutions - including major banks, media outlets and airlines - have reported suffering a mass IT outage.
The US state of Alaska has warned its emergency services are affected, while several of the country's airlines have grounded their flights around the globe.
Australia - which has been particularly hard hit - has seen broadcast networks scrambling on air as systems failed and supermarkets crippled. Sky News UK went completely off air as a result of the issues.
The cause of the outage is unclear, but many of those impacted have linked it to Microsoft PC operating systems.
An official Microsoft 365 service update posted to X earlier in the day said " we're investigating an issue impacting users ability to access various Microsoft 365 apps and services".
However, a Microsoft spokesperson told the BBC on Friday that "the majority of services were recovered" hours earlier.
A spokesperson for Australia's Home Affairs Minister said the outage appears to be related to an issue at global cybersecurity firm Crowdstrike, and the country's cybersecurity watchdog said there is no information to suggest it an attack.
"Our current information is this outage relates to a technical issue with a third-party software platform employed by affected companies," they said in a statement.
Alaskan officials said many 911 and non-emergency call centres are not working properly.
United, Delta and American Airlines - which are all based in the United States - have issued a "global ground stop" on all of their flights. And in Australia, carriers Virgin Australia and Jetstar have also had to delay or cancel flights.
Australian telecom firm Telstra has said triple-0 call centres - the main emergency contact in the country - are not affected, but that it is working with other state emergency services providers to implement backup processes.
Social media users have reported queues at Australian stores like Woolworths, with payment systems downed, and trouble accessing financial institutions like the National Australia Bank.
Apparently this is causing quite a fuss in a lot of places. Airports screwed up and that sort of thing.
I find it interesting as the whole discussion about data centres and the move to the cloud pops up all the time.
In theory it all makes sense. Instead of investing tonnes of money into building your own data services, your own security, and all this other really specialised stuff that is completely outside of your company's domain, you instead hire a company that is actually a specialist to do it.
Its do you build your own vault or just put your money in the bank.
But...a few years ago I was working at a large (non-tech) multinational who were investing big money to construct a big on-premise server centre at their global HQ. This was after the mass move to cloud was well underway and they chose a different path.
What we're seeming to see here is the danger of a single point of failure. What seems to be a pretty minor technical fault at Microsoft causing big issues globally.
An interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Thoughts?
I am not saying this invalidates your point or it's not worthy of discussion, but the current mess is because of CrowdStrike which is -apparently- a very popular security software. Seems like everyone who was using it got nuked by a fucked up update.
Quote from: Tamas on July 19, 2024, 03:59:41 AMI am not saying this invalidates your point or it's not worthy of discussion, but the current mess is because of CrowdStrike which is -apparently- a very popular security software. Seems like everyone who was using it got nuked by a fucked up update.
I don't know why my router was fucked up for the last two day. I think it was because of the DNS server I use. Probably linked to CrowdStrike.
Yeah sounds like it was the CrowdStrike thing - there was an issue with Microsoft but it was fixable (patch a few weeks ago apparently). But obviously some organisations with poor patching are experiencing both.
Separately can't help but think somewhere someone is sitting who pressed go on that software update. Obviously it's not their fault - it's a system issue - but they're watching this knowing they pressed the button and just have that sinking feeling of the most massive work fuck up :lol: :(
Interesting to see some journalists (not tech journalists) being absolutely baffled by this - stuff like "how can a software fault from a third party basically shut down airports all around the world?" I feel like this might be a moment (a bit like the supply chain issues post-covid) where reporters in unrelated areas, including politics, start realising that the world we live in is a little different than they'd understood and these areas (all of them in a way supply chains - digital and physical) are really important.
Edit: Eg this timelapse of flights over the US:
https://x.com/US_Stormwatch/status/1814268813879206397
If you're using a Microsoft product and don't expect it to crash, then it's on you.
The Crowdstrike issue today was not related to cloud computing. Instead it caused a blue screen of death on your local Windows client. Non-Windows clients (iOS/MacOS , Android, Linux...) and non-Windows servers (i.e. most of "the Cloud") weren't affected.
Wondering what will happen when I turn on my work laptop in a little while....
Quote from: Jacob on July 19, 2024, 10:46:57 AMWondering what will happen when I turn on my work laptop in a little while....
In my workplace it was gone by 8am CET, only machines running before that were affected.
Quote from: Josquius on July 19, 2024, 03:19:38 AMI find it interesting as the whole discussion about data centres and the move to the cloud pops up all the time.
In theory it all makes sense. Instead of investing tonnes of money into building your own data services, your own security, and all this other really specialised stuff that is completely outside of your company's domain, you instead hire a company that is actually a specialist to do it.
Its do you build your own vault or just put your money in the bank.
But...a few years ago I was working at a large (non-tech) multinational who were investing big money to construct a big on-premise server centre at their global HQ. This was after the mass move to cloud was well underway and they chose a different path.
As others have noted, this problem is related to Windows and some particular Microsoft software-as-a-service (SaaS) offerings. Azure, Microsoft's cloud infrastructure offering, is unaffected (as evidenced by Languish still being up). I could go on a tangential rant about cloud migrations and the false dichotomy between "run it all In The Cloud™" and "build your own datacenter from scratch", but I won't.
QuoteWhat we're seeming to see here is the danger of a single point of failure. What seems to be a pretty minor technical fault at Microsoft causing big issues globally.
Technology single points of failure have been an endemic problem in business for decades. The root cause isn't a particular technology, it's the attitude of the business. Businesses, like people, can't deal with very unlikely but existential risk. If a particular cloud vendor, SaaS provider, or sole supplier looks to be very low risk it's very difficult to get the business to care about building in redundancy or developing contingency plans. Those things cost money, both directly and indirectly though slowing down the rate that money-making value can be delivered to customers.
Of course, this is a multi-layered problem. The customers of these businesses also generally don't appreciate the very low but existential risk that could ripple down the chain to them, and so don't value businesses that account for this enough to pay extra for the product or service. The result is a fragile system where one key supplier having a problem cascades through a web of interconnected businesses, causing something like what the article describes. This time it happened to include a SaaS product, but there is nothing unique in such products that would cause this.
QuoteAn interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Stadia had two promises. One was giving the customer much better and more frequently-updated hardware than they may have at home on which to play games; the other was giving the customer an extensive rental library of games so that they don't have to buy as many. I don't think it was a bad idea
per se, but I think Google significantly overestimated the addressable market. While I generally agree with you about preference for local hardware, there is a market for those who can't afford the latest console or a tricked-out gaming PC, or those who travel a lot. It's just not a large enough market for such an expensive to run service, especially since there are services like Microsoft Game Pass that provide the second value proposition (and work on existing hardware, for those who have it).
Been working on this since 5 am... it caused mass chaos across all of our systems
It's the second (and by far the biggest) fuckup of Crowdstrike in two weeks.
Two Fridays ago everything slowed to a crawl at my work because of another failed update (simply opening a file took like a minute).
Palo Alto Networks about to make bank! (another EDR provider)
(https://i.redd.it/4ax14cdm6gdd1.jpeg)
Or, did the AI corrupt a CrowdStrike update? :ph34r:
I, for one, welcome our new silicon-based overlord.
Quote from: Josquius on July 19, 2024, 03:19:38 AMAn interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Thoughts?
Chromebooks and low spec computers are cheap and plentiful, high spec computers are big and expensive or less big and really expensive. The AAA gaming ecosystem is built around high-cost hardware and planned hardware obsolescence, a nasty combination. Cloud gaming can liberate the user from the GPU upgrade treadmill, but even more importantly, it liberates the user from dedicated hardware, period. You can play on any device (screen) you want in virtually any form factor.
Yes fast and reliable internet connections aren't available everywhere in the world, but where they are available it makes that service attractive. And the value and use cases of building out network infrastructure go well beyond gaming so investment will continue to pour in.
Quote from: The Minsky Moment on July 19, 2024, 02:36:54 PMQuote from: Josquius on July 19, 2024, 03:19:38 AMAn interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Thoughts?
Chromebooks and low spec computers are cheap and plentiful, high spec computers are big and expensive or less big and really expensive. The AAA gaming ecosystem is built around high-cost hardware and planned hardware obsolescence, a nasty combination. Cloud gaming can liberate the user from the GPU upgrade treadmill, but even more importantly, it liberates the user from dedicated hardware, period. You can play on any device (screen) you want in virtually any form factor.
Yes fast and reliable internet connections aren't available everywhere in the world, but where they are available it makes that service attractive. And the value and use cases of building out network infrastructure go well beyond gaming so investment will continue to pour in.
Cloud gaming has too much latency for most games to be played reliably on ultra high graphics, it requires lowering graphic quality.
It's good for certain types of games, not good for others.
It's always going to be a niche product, never be of mass appeal like PC or console. That is, until we can fix the problem with very high speed internet connections being available everywhere and not just for a tiny portion of the population (upload and download)
Interestingly, our school (which runs quite a bit using Microsoft platforms) chose today tell everyone to stay home and run a "virtual class" stress test. :lol:
However, we also mostly use MacOS operating systems...and had no apparent issues.
Today's outage was way less about cloud versus on-prem and way more about letting go of the reins on software updates.
Best practice used to be "download the update and try it out on a canary system, then only release it into the wild after seeing it didn't make the canary's organs liquidate and pour out of every orifice."
Then the bean counters started whining about things like opportunity cost and time to market, and now all the expectations are to release things so fast that you only notice the dead canary briefly as a blurry image now that your eyeballs are turning to goop, too.
PS - Azure was so heavily affected because a lot of "platform as a service" offerings just spin up a Windows virtual machine under the hood anyway, frequently from a template supplied by the customer, which may very well include the Crowdstrike Falcon sensor rolled into the base image because SECURITAH. Take it from someone who has operated an Azure tenant that cost $20mil in a month (we had to work to get it down to $8mil/mo- Microsoft has gone on record saying we have one of the biggest tenants in their cloud and have a hard time keeping up with it because of ALL the resources we're running).
Quote from: Tonitrus on July 19, 2024, 06:02:57 PMInterestingly, our school (which runs quite a bit using Microsoft platforms) chose today tell everyone to stay home and run a "virtual class" stress test. :lol:
However, we also mostly use MacOS operating systems...and had no apparent issues.
So... it was planned :tinfoil:
Quote from: DontSayBanana on July 19, 2024, 08:20:42 PMToday's outage was way less about cloud versus on-prem and way more about letting go of the reins on software updates.
Best practice used to be "download the update and try it out on a canary system, then only release it into the wild after seeing it didn't make the canary's organs liquidate and pour out of every orifice."
Then the bean counters started whining about things like opportunity cost and time to market, and now all the expectations are to release things so fast that you only notice the dead canary briefly as a blurry image now that your eyeballs are turning to goop, too.
PS - Azure was so heavily affected because a lot of "platform as a service" offerings just spin up a Windows virtual machine under the hood anyway, frequently from a template supplied by the customer, which may very well include the Crowdstrike Falcon sensor rolled into the base image because SECURITAH. Take it from someone who has operated an Azure tenant that cost $20mil in a month (we had to work to get it down to $8mil/mo- Microsoft has gone on record saying we have one of the biggest tenants in their cloud and have a hard time keeping up with it because of ALL the resources we're running).
This is a good assessment of what happened.
I have noticed at my company QA is getting sloppier and sloppier too, and management pressures our developers to get out enhancements "NOW NOW NOW", because.... well, "because" :hmm:
As more of the systems I manage get forced into clouds and onto SaaS platforms, we no longer have much of an opportunity to QA anything; the vendor supposedly does it but I don't honestly believe our primary ERP vendor has a real QA team at all anymore, given how sloppy and bug-ridden most of their rollouts have become.
Quote from: viper37 on July 19, 2024, 05:37:36 PMQuote from: The Minsky Moment on July 19, 2024, 02:36:54 PMQuote from: Josquius on July 19, 2024, 03:19:38 AMAn interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Thoughts?
Chromebooks and low spec computers are cheap and plentiful, high spec computers are big and expensive or less big and really expensive. The AAA gaming ecosystem is built around high-cost hardware and planned hardware obsolescence, a nasty combination. Cloud gaming can liberate the user from the GPU upgrade treadmill, but even more importantly, it liberates the user from dedicated hardware, period. You can play on any device (screen) you want in virtually any form factor.
Yes fast and reliable internet connections aren't available everywhere in the world, but where they are available it makes that service attractive. And the value and use cases of building out network infrastructure go well beyond gaming so investment will continue to pour in.
Cloud gaming has too much latency for most games to be played reliably on ultra high graphics, it requires lowering graphic quality.
I use GeForce now and I've never had an issue playing at home. At highest settings.
Quote from: The Minsky Moment on July 22, 2024, 02:56:23 PMQuote from: viper37 on July 19, 2024, 05:37:36 PMQuote from: The Minsky Moment on July 19, 2024, 02:36:54 PMQuote from: Josquius on July 19, 2024, 03:19:38 AMAn interesting argument I heard the other day about why AI is just a fad came onto Google Stadia- that cloud gaming is completely backwards thinking. Computers are cheap and plentiful. Its reliable ultra fast internet that is a rare commodity in the world. For most people it will never make more sense to play a game on a computer hundreds of miles away rather than just hooking one up to your TV.
Thoughts?
Chromebooks and low spec computers are cheap and plentiful, high spec computers are big and expensive or less big and really expensive. The AAA gaming ecosystem is built around high-cost hardware and planned hardware obsolescence, a nasty combination. Cloud gaming can liberate the user from the GPU upgrade treadmill, but even more importantly, it liberates the user from dedicated hardware, period. You can play on any device (screen) you want in virtually any form factor.
Yes fast and reliable internet connections aren't available everywhere in the world, but where they are available it makes that service attractive. And the value and use cases of building out network infrastructure go well beyond gaming so investment will continue to pour in.
Cloud gaming has too much latency for most games to be played reliably on ultra high graphics, it requires lowering graphic quality.
I use GeForce now and I've never had an issue playing at home. At highest settings.
You are in New York City. Not living in a rural road on the countryside with shitty upload speed and limited download speed. max I can get is 200/50 and I have to buy a business internet subscription for that. Otherwise, I'd have 100/30 with throttling. And they tell me they have no plans to upgrade anytime soon.
Quote from: viper37 on July 23, 2024, 07:26:28 PMYou are in New York City. Not living in a rural road on the countryside with shitty upload speed and limited download speed. max I can get is 200/50 and I have to buy a business internet subscription for that. Otherwise, I'd have 100/30 with throttling. And they tell me they have no plans to upgrade anytime soon.
Yeah that would definitely be an issue for cloud gaming. Here bandwidth is plentiful but space is at a premium. So cloud gaming >> big gaming rig.
Quote from: The Minsky Moment on July 24, 2024, 12:52:41 AMQuote from: viper37 on July 23, 2024, 07:26:28 PMYou are in New York City. Not living in a rural road on the countryside with shitty upload speed and limited download speed. max I can get is 200/50 and I have to buy a business internet subscription for that. Otherwise, I'd have 100/30 with throttling. And they tell me they have no plans to upgrade anytime soon.
Yeah that would definitely be an issue for cloud gaming. Here bandwidth is plentiful but space is at a premium. So cloud gaming >> big gaming rig.
Yeah. I can see the appeal for city centre.
But my cousin is in Montreal, not too far off the city center, and he can't get fiber.
I see the need for both high upload and download speed when using cloud gaming as nothing is local. Cable internet can have decent download speed, but you are often stuck at low (<50mbps) download speed, and that is still a vast majority of the occidental world. Not to mention other remote parts that get their internet with satellites.
Delta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss (https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss)
We'll see how far that goes.
Quote from: viper37 on July 31, 2024, 05:01:06 PMDelta CEO: Company Suing Microsoft and CrowdStrike After $500M Loss (https://www.thedailybeast.com/delta-ceo-says-company-suing-microsoft-and-crowdstrike-after-dollar500m-loss)
We'll see how fare that goes.
Let's see how strong that user contract is.