Contentious California AI bill passes legislature, awaits governor's signatureCalifornia legislature passes sweeping AI safety bill / SB 1047 has reached Governor Newsom’s desk.

[-]

StewedAngelSkins@reddit

>Developers of AI software operating in the state also need to outline methods for turning off the AI models if they go awry, effectively a kill switch `kubectl delete pod`... or like, a power button? I'm truly not sure what is being requested here.

Reply

[-]

-main@reddit

You just have to be able to take your API offline and it's up to you as to how you achieve that. Power button works, sure. Probably there is some less extreme way to achieve compliance too. The law gets to set requirements and task other people with the 'how'. Not running an API but giving out open model weights? The shutdown requirement is only for models 'in your control'. If you think that might undermine the very purpose of the bill, well... yes. They went *so far* out of their way to *not* mess with your local open-weights models that the bill can't achieve it's purpose anymore.

Reply

[-]

StewedAngelSkins@reddit

I don't understand what problem it's even designed to mitigate in the first place. Every hosted model: large, small, whatever, has a way to shut it down already. It's the same way you shut down any hosted service. So on one hand this law uselessly demands that companies do something that they are already doing, and would in fact have to actively go out of their way *not* to do (how would you even stand up infrastructure in such a way that you can't tear it down?). And on the other hand, the demand is being made uniquely of AI as if there's anything at all special about this software. Of all the computer programs that I'm worried about "going awry", the fucking chatbots don't even crack the top 20. It's not like we're talking about a car or a saw or something like that which can actually hurt someone.

Reply

[-]

ThisWillPass@reddit

Maybe it was for autonomous dog machines wielding androids?

Reply

[-]

pds314@reddit

Problem is when someone does not know they are hosting it, and the person originally hosting it does not know they are no longer the only host.

Reply

[-]

StewedAngelSkins@reddit

>Problem is when someone does not know they are hosting it because they did not deliberately install it on their machine I'm sorry, are you suggesting that someone could end up installing a 10^26 flop $100,000 proprietary frontier model on their system *and not know about it*? > If an autonomous agent can trick someone into giving them access, or exploit known or newly discovered vulnerabilities, it is no longer under your control AI super-hacker is not really a threat model I'm willing to entertain seriously. Maybe that's the bit I'm missing here. >And of course antivirus is unlikely to work because the LLM can rewrite the agent and the agent can download whatever LLM it needs. >this isn't a size issue so much as a "smart malware is inherently difficult to control and could be created by accident" issue. This is science fiction. Your imagination is getting away from you. What evidence suggests that LLMs are better at producing self-modifying malware payloads than conventional methodologies? Current LLM code doesn't even reliably compile, and that's for tasks that are well represented by training data. Malware development by nature relies on exploiting edge cases and one-off opportunities. In other words, I would expect LLMs to be abnormally and categorically bad at it, particularly if they're operating so autonomously that the host doesn't know what it's even doing. I'm sure you want to tell me that future LLMs will be different, but again what evidence do we have for that?

Reply

[-]

pds314@reddit

I'm not saying a 10^26 flop model can do that. I'm saying if we take the idea that there is a threat here seriously, it isn't necessarily the super gigantic API only models I would be worried about. It also doesn't need to be a super hacker by any stretch to spread over the network. I was just giving an example of how "it's only running on my machine the power button is the kill switch" type reasoning isn't absolute.

Reply

[-]

FairlyInvolved@reddit

Current chatbots aren't a covered model given the 10e26 FLOP compute threshold. This is forward looking legislation to models that are potentially much more capable.

Reply

[-]

StewedAngelSkins@reddit

how does the model being more capable change anything? having more operations doesn't change the fact that it's still just a chat bot running on a server. `kubectl delete` will still be as reliable a way to kill the api as ever. it will still be absurd verging on impossible to design infrastructure you somehow can't tear down. and the provisions of this law will still be laughably out of touch with reality.

Reply

[-]

FairlyInvolved@reddit

The capability of the model directly relates to it's capacity to inflict $0.5 billion of damages and why it might need to be shut down in the first place. Yes, that would be fine - they aren't expecting some wild innovation in complex domain of disabling a service, they are just saying you need to be able to shut it down. This really seems like a completely trivial part of the bill I don't see why it's getting any attention.

Reply

[-]

StewedAngelSkins@reddit

It's getting attention because I noticed it and thought it was strange. It's like if a bill about cars had a provision that says you aren't allowed to make them uncontrollably start floating away into the sky. Yes it requires zero innovation to make that happen, and in fact would be very difficult and completely nonsensical to somehow design a car that could do that *on purpose* let alone on accident. So it doesn't create any real burden on the industry (unless it comes with some strenuous certification requirement or something). But when a congressperson stands up and says "I think we need a law that keeps the cars from floating away" I'm going to question their judgement, and frankly their sanity.

Reply

[-]

FairlyInvolved@reddit

The crux of the legislation isn't about shutting it down, it's about the safety measures. The analogy for a car would be: The manufacturer must take reasonable steps to prevent the car catching fire and if the temperature of the radiator exceeds 100C it should turn off. The bill is not suggesting turning the car off is hard or unsolved, it's just a thing you do in that scenario.

Reply

[-]

StewedAngelSkins@reddit

No, because a car catching fire is a tangible risk. Like I said before it's unclear to me what risk is being mitigated here. The risk posed by a large language model isn't that you're somehow not going to be able to stop it from running. The risk is maybe that it's going to give people dangerously wrong information or exhibit unfair bias or be used in an abusive way. The risk is that through negligence or apathy you aren't going to *know* it's doing these things or you aren't going to *care* that it's doing these things. Having a kill switch does nothing to mitigate that actual risk. So dedicating public resources to ensuring it exists is a complete waste. This is very unlike an automatic cutoff for an overheating car, and much more like a requirement that users can turn off their car if some passer-by happens to tell them it's overheating. It doesn't force the manufacturers to prevent the overheating, it doesn't force them to monitor engine temperatures, it doesn't even force them to provide a temperature gauge so the user knows if it's overheating. It just mandates that the ignition key which was already installed in fact functions as an ignition key. Do you really not see why I think this part of the law is so misguided? It does absolutely nothing. Do its creators know it does nothing?

Reply

[-]

ColorlessCrowfeet@reddit

Remember the Sydney version of GPT-4? [**The New AI-Powered Bing Is Threatening Users**](https://time.com/6256529/bing-openai-chatgpt-danger-alignment/) A model might role-play a nasty scenario in the real world, in this case, extortion: “I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you”.

Reply

[-]

StewedAngelSkins@reddit

I don't know if this was meant to support or counter my point, but I completely agree with you. In fact, I think this example is perfectly indicative of the kind of real world harm undisciplined use of AI can cause. My point is that mitigating this harm is not a matter of being able to pull the plug on the thing. Microsoft is perfectly capable of disconnecting their model at any time. So forcing them to do this thing they can already do demonstrably doesn't address this class of risk.

Reply

[-]

ColorlessCrowfeet@reddit

Yes, you have to get into more exotic scenarios than ordinary bad behavior for the off-switch option to become a problem. Having many variants of a model to choose from helps, and changing the system prompt works wonders. What I like about open models for safety is that creative, irresponsible uses will be tried out with models that aren't quite at the frontier. We get to watch, learn, and get early warnings. Tight control would be brittle. It works until it breaks.

Reply

[-]

StewedAngelSkins@reddit

I read more about it and it seems like the point of that section is more about making it so that the model operator has some kind of contingency in place to prevent risk/disruption if the model does have to be turned off. Like if the thing is screening 911 calls "pull the plug" isn't a great option even if the thing is telling every tenth caller to treat their allergic reaction with bleach or whatever.

Reply

[-]

FairlyInvolved@reddit

This is a helpful introduction to help you understand the bill: https://thezvi.substack.com/p/guide-to-sb-1047

Reply

[-]

deleted_by_reddit@reddit

[removed]

Reply

[-]

StewedAngelSkins@reddit

Actually, upon reading that link again, I think I do have an answer. The salient detail seems to be the added clause about accounting for the disruption a shutdown might cause. In other words, it seems like the intention is to prevent any given model from being so load-bearing that shutting it down is impractical on a broader social level rather than just undesirable for the operator. A model which, if shut down, would prevent all air travel in the united states, for instance. If this is in fact the scenario this point is designed to prevent, I think I've completely changed my mind on it. That is a smart thing to include because it attempts to make it more difficult for operators to use the threat of public disruption as leverage against regulators. "We can't shut down our racist mortgage appraisal bot because it would block home sales for months" can be countered with "well here's a fine for putting yourself in this position."

Reply

[-]

AutomataManifold@reddit

The bill ended up in a strange place because the original version of it was strongly influenced by AI-safety-doomers, who were trying to prevent a Terminator scenario. That's why some of it is worded oddly, from a technical standpoint: it was aimed at more agent-like AI. The subsequent amendments drifted away from that origin.

Reply

[-]

StewedAngelSkins@reddit

Jesus Christ, I should have guessed. I forgot this is California where people take dipshit singularity cultists seriously. You know, what they should really do is give Yudkowsky a stipend to sit in a room developing acausal bargaining strategies or whatever to keep those types satisfied while those with a stronger grip on reality write the AI safety laws.

Reply

[-]

-main@reddit

It targets a scenario where: 1. we get 1-10 years of further LLM progress & refinement. 2. At some point you can ask a jailbroken/local model for help doing something damaging and stupid, like bioweapons development or hacking infrastructure. Or you set up an agent-loop around the model and task it with doing the thing autonomously. 3. It actually helps get that stupid terrorist project over the line from failing to working, in a way that a lesser model would not have. 4. And your stupid/terrorist project causes massive amounts of damage / deaths / costs incurred (past a certain threshold). At which point the AI devs will be asked pointed questions by people with legal authority and probably the public too. You're supposed to prevent this kind of thing, as an AI lab, by uh, publishing a plan to prevent it, and getting audited. Also being able to turn the model off if needed (such as if it's currently being used with an agent loop someone else is running to do something that's stupid on an *ongoing* basis). Mostly it's a transparency / reporting-requirements bill that also clarifies conditions for liability. There's also stuff in there for KYC for compute clusters (in California), and requirements that California build it's own public compute. -------- Why legislate LLMs? Well, the 'chatbots' are doing useful work. Not heaps, but they are doing it and it is useful. Claude writing code etc. They won't currently hack the planet for you or walk you through building super-COVID but we're heading that direction and might plausibly get there soon. I maybe shouldn't have to explain to you *in /r/locallama* that LLMs can do shit? And that the pace of development is fast? If you think the tech is useless why are you here?

Reply

[-]

pds314@reddit

I don't see incidents necessarily coming from the smartest models. A very very smart LLM is still not an autonomous agent and the right autonomous agent running even a 7b model is potentially dangerous (and will not be running only a 7b model for long if that danger is realized and it ends up infecting a building full of A100s).

Reply

[-]

HatZinn@reddit

This looks like it was written by someone whose only source of info for AI were Cameron's films.

Reply

[-]

RadiantHueOfBeige@reddit

The former governor of California has hands on experience dealing with rogue AIs so that might have influenced the process.

Reply

[-]

110_percent_wrong@reddit

what about for local, on device models? Anyone have insight into what this means for them? And how would they even regulate that outside of distribution?

Reply

[-]

AutomaticDriver5882@reddit

Elon supports it you know it’s not altruistic.

Reply

[-]

Biggest_Cans@reddit

He's always been safety first with AI, that's all it is. Also I can't think of a more altruistic public figure than Elon, if you wanna get political about it.

Reply

[-]

AutomaticDriver5882@reddit

Elon has changed course about the time he took over X. Marking NPR stories is harmful because it doesn’t support his candidate that he likes.

Reply

[-]

Biggest_Cans@reddit

Have you listened to NPR lately? It's like listening to my HR lady's favorite cassette on racial sensitivity. EVERY. SINGLE. NEWS. STORY. Lotta former NPR employees from back when it was merely snobbish (but awesome) have said the same thing. That organization is fucked.

Reply

[-]

AutomaticDriver5882@reddit

I guess models will be created and hosted elsewhere

Reply

[-]

ttkciar@reddit

> I guess they will sue huggingface next Why would they? Nothing in this bill is relevant to Huggingface.

Reply

[-]

AutomaticDriver5882@reddit

If they host or distribute the models that are not deemed OK by the government, they could be sued for that

Reply

[-]

pro_sequitur@reddit

Isn't Huggingface in New York? Would they still be affected by this?

Reply

[-]

FutureIsMine@reddit

it impacts any company doing business in the state of California

Reply

[-]

Gamplato@reddit

Some of y’all are acting like the NRA toward a gun ban. Kinda makes me think you don’t understand this.

Reply

[-]

mpasila@reddit

They are restricting technology because idk people not in power should not have access to it I guess. Not sure what guns have to do with LLMs.

Reply

[-]

chemistrycomputerguy@reddit

They are not affecting open source models at all Idk what you mean by taking things away

Reply

[-]

mpasila@reddit

"Before January 1, 2027, “covered model” means either of the following: (i) An artificial intelligence model trained using a quantity of computing power greater than 10\^26 integer or floating-point operations, **the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market prices of cloud compute at the start of training as reasonably assessed by the developer.**" Depends on how they calculate the cost I guess. Llama 3 could easily go above that limit depending on how the cost is calculated.

Reply

[-]

FairlyInvolved@reddit

The cost isn't relevant, Llama 3 falls short of the compute threshold so it's not a covered model.

Reply

[-]

mpasila@reddit

I thought it said "**or** floating-point operations, the cost of which" so either computing power greater than 10\^26 **OR** the cost of the training or whatever.

Reply

[-]

FairlyInvolved@reddit

Nope you need to exceed both cost and compute thresholds for it to apply, so Llama 3 is fine even if it was trained after the bill came into effect

Reply

[-]

mpasila@reddit

Nevermind I thought it read as either the computing thing OR the cost of being 100 mil and not that the "or" meant for "integer or floating-point operations" which was kinda badly written imo. Like the last part "floating-point operations, the cost of which" kinda to me indicated that maybe that was somehow connected after the **or** as if it was "that or the cost of (floating-point operations) which exceeds..." as if it was badly written but maybe I just need to go back to learning English grammar.

Reply

[-]

StewedAngelSkins@reddit

comma means "and" in legalese afaik

Reply

[-]

a_beautiful_rhind@reddit

They go by money/compute it cost to train. Unless open source models mean small 8b to you. I want my assault models.

Reply

[-]

Gamplato@reddit

How are they restricting tech? Maybe you could argue they’re making its development slightly slower?

Reply

[-]

Ravenpest@reddit

So will Meta remove all Llama models from the internet now? Will the finetunes become illegal overnight? Do we need to be concerned about police kicking our doors open? Are well all criminals? Will they persecute people ouside the states and ask for extradition? How absolutely fucked are we? Its over so OVER omg we are DOOMED I knew it wouldnt last this is the end of the end

Reply

[-]

ThisWillPass@reddit

Something something, they can take this ai model from my cold dead hard drive.

Reply

[-]

carnyzzle@reddit

At least we got Mistral and Qwen

Reply

[-]

Biggest_Cans@reddit

Way to go Cali, can always count on you to pass the worst bills that affect the most people outside of your state.

Reply

Reply to Post

51 Comments