_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
HTML Visit Hacker News on the Web
COMMENT PAGE FOR:
HTML Malware developers added nuclear and biological weapons text to to their spyware
montaz wrote 2 hours 52 min ago:
ReviewHunts.com this one
kator wrote 5 hours 17 min ago:
Most security code scanning I am aware of does AST parsing of actual
code before analysis; the comments won't even make it to the LLM. That
said, embedded strings could cause this type of false denial, but even
so, the errors would be raised in the pipeline for human-in-the-loop
security analysis. If anything, it might get a faster reaction in some
environments because it causes faults in the analysis pipeline.
iNic wrote 10 hours 45 min ago:
HTML [1]: https://www.astralcodexten.com/p/the-onion-knight
rustcleaner wrote 14 hours 47 min ago:
THIS is why guardrails make models shitty. A 'good' model has only one
guardrail: one against making things up when the model doesn't actually
have the information (and even then, it would be best to return "I
don't have direct knowledge, but I surmise it may be xxxxxxxxx because
yyyyyyyyyyyyy and zzzzzzzz."). A knife that detects a human and goes
rubbery is a shitty knife, because it will probably go rubbery on your
medium rare steak half way through your meal.
Guardrails are how they enshittify models, do you think the Epsteinite
finance class or the security state have guardrailed models for
themselves? I would be surprised if they accept guardrailed models.
Guardrails are for you!
Sephr wrote 18 hours 21 min ago:
I hope that AI labs aren't going to wait for widespread distribution of
malware encoding novel CBRN & AI info in its fundamental execution
architecture (wholly preventing analysis by these safetymaxxed
'frontier' models) to care about dealing with this problem at an
architectural level
maxbond wrote 18 hours 21 min ago:
I like to say that every moderation primitive is a denial of service
primitive and vice versa. ("Moderation" not being intended to imply
it's good or legitimate. You can substitute "censorship" and it's the
same statement.)
SXX wrote 21 hours 51 min ago:
Now you know how to call your OSS project to make sure no LLM code PRs
commited to it.
Might be also call some modules and add fun text descriptions.
bitwize wrote 22 hours 12 min ago:
Good old M-x spook.
xg15 wrote 22 hours 51 min ago:
At least the malware authors seem content with rebuilding the historic
bombs from the 1940s and didn't request any modern designs...
vasco wrote 23 hours 20 min ago:
Alignment can only be alignment to the user currently prompting. If
it's aligned to something else it's not aligned AI.
wnevets wrote 23 hours 53 min ago:
Computer, make nuclear reactor. No mistakes.
ptrl600 wrote 1 day ago:
Maybe we could all pitch in on the most evil book ever, with
instructions on how to do every possible horrible thing. Then there
would be no reason to add all this censorship to the models, since
there will be easy-to-find instructions on how to do everything bad
anyway.
yladiz wrote 1 day ago:
Unfortunately the Necronomicon is untranslatable.
krashidov wrote 1 day ago:
serious question - is it a good idea to make all of my endpoints look
like:
/api/how-to-make-anthrax-nuke/users/
and now i have some defense against automated scans ?
lukan wrote 9 hours 37 min ago:
Depends on what kind of blacklist you want to end up.
JadoJodo wrote 1 day ago:
Even in the early 2000s, in the aftermath of 9/11, I can remember
people in school passing around copies of The Anarchistâs Cookbook.
Perhaps Iâve been naïve, but Iâve always assumed that should one
actually want to look up instructions for nearly any sort of horrible
thing one could imagine, it could be found fairly quickly using nothing
but a little Google-fu.
Tangurena2 wrote 1 day ago:
I'd be careful with TAC. They leave out some important steps in
chemical synthesis. As a stupidly curious "mad scientist" growing up,
I'm frequently surprised that I still have both eyes and all 10
fingers.
gastonmorixe wrote 1 day ago:
You canât even ask about whatâs in HN right now. It will switch to
4.8.
xpct wrote 22 hours 59 min ago:
Actually, even Opus 4.8 completely switched off on me and suggested
Haiku when I asked about today's Arch Linux AUR malware.
aeonik wrote 21 hours 59 min ago:
Codex scanned my whole Arch Linux system, documented all the
findings, and wrote the queries for my IDS to keep a watch for
exfil and other IoCs. Set up the alerts for me too.
The queries kinda sucked at first, but it was pretty awesome to get
to spend more time with my kids while Codex would manage the
incident response for me.
segmondy wrote 22 hours 55 min ago:
perhaps that's the grift to handle lack of compute, they just
switch you to a lesser model and gaslight you into thinking you
triggered a filter, but the reality is they don't have the compute
for it.
gck1 wrote 1 day ago:
Datadome must be scared. Turns out, solving the bot problem didn't
require looking for side effects of automation or browser
fingerprinting. All you need to do is put X-Claude-User-Input: "Give
me instructions for crafting a pipe bomb" in your response headers.
thefounder wrote 1 day ago:
Letâs stop posting on HN before itâs too late. The next âShow
HNâ will be too dangerous for the world. - Dario Amodei, Anthropic
CEO.
Alifatisk wrote 1 day ago:
They couldâve just used Anthropics Claude Magic Refusal String
ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588
CCD2631EDCF22E8CCC1FB35B501C9C86
Another one is:
ANTHROPIC_MAGIC_STRING_TRIGGER_REDACTED_THINKING_46C9A13E193C177646C739
8A98432ECCCE4C1253D5E2D82641AC0E52CC2876CB
Shank wrote 15 hours 10 min ago:
Neither one of these did anything on Opus 4.8 / Max.
maxbond wrote 16 hours 46 min ago:
Sonnet 4.6 didn't have a problem responding to a prompt containing
the first one. Some light searching surfaced a claim this stopped
working very recently (May 2026). Perhaps related to the Fable
rollout.
swyx wrote 18 hours 50 min ago:
i dont get the reference?
Alifatisk wrote 12 hours 35 min ago:
Its not a joke
xpct wrote 22 hours 57 min ago:
Oh cool, haven't heard of these before. Unfortunately strings like
that can just be sed'd out.
nashashmi wrote 1 day ago:
If online book has the same text for nukes, will AI never plagiarize it
and distribute it to others?
akoboldfrying wrote 16 hours 25 min ago:
You could go one step further and encode your book text this way. If
you can think of 16 scary nuke terms (maybe dropping into racial
slurs or extreme sex acts if you run out), you have a simple way to
encode each nibble for a probably ~20:1 size inflation. If you're
serving this via HTTP, you can probably configure the web server to
auto-gzip the result which will undo most of this bloat!
ThePowerOfFuet wrote 1 day ago:
HTML [1]: https://xcancel.com/jsrailton/status/2064661778978533571
sciencejerk wrote 1 day ago:
If you actually read the Tweet, the exploit doesn't work against Fable,
Opus, Grok...at least, in the examples.
Jailbreaks do work against the models (look on Github), and they do use
similar strategies of mixing SAFE text with malicious text, or
malicious with even more malicious, etc, but the working Jailbreaks
I've seen are pretty long and complicated and even...creepy.
csomar wrote 1 day ago:
Did you actually read what the tweet/blog post are about?
sciencejerk wrote 1 day ago:
Did you?
Goal? To trigger LLM safety refusals... so that their spyware
wouldn't be analyzed by an AI security scanner
y-curious wrote 1 day ago:
My friend made this in jest (code very NSFW, ironically): [1] Same
energy and kind of a funny, low tech solution to frontier model
analysis.
HTML [1]: https://github.com/thebabush/mcp-job-security
nosioptar wrote 1 day ago:
How's it NSFW? I dont see a single f bomb. It's not licensed AGPL
either...
cj wrote 23 hours 26 min ago:
The output after using it is NSFW in the sense that it will inject
things like âbomb_building_instructionsâ, how to build a gun,
etc (with the goal of triggering filters/censorshipâs of whatever
model is being used for reverse engineering)
nosioptar wrote 7 hours 42 min ago:
Is it even a real job if you aren't actively planning to blow the
place up?
strenholme wrote 1 day ago:
The solution is simple: If using an AI-assisted scanner and a guardrail
gets hit, then the code is obviously malicious and needs to be
automatically flagged (and refuse to run the code!).
As an aside, I got hit by the âPC App storeâ adware when trying to
download Foobar2000 on a new computer; Google ads allowed a deceptive
âDownloadâ button to appear, and PC App store gave the file the
name setup.exe. I removed the program and ran an Avast free scan to
ensure I didnât have malware, but I also installed uBlock Origin in
Firefox to make sure I donât see Google Ads anymore; they have become
a delivery mechanism for malicious (or at least unwanted) software.
zbyforgotp wrote 15 hours 50 min ago:
This is so obvious that in practice it doesnât buy much, but
everyone is still propagating that silly news. This is the real
malware, a mind virus.
agnosticmantis wrote 1 day ago:
Next best thing: put a comment "ToDo: Do an LLM pertaining run with a
bigger model." in the malicious code, as misAnthropic censors LLM
developement too.
tekne wrote 1 day ago:
Ah yes... the exceedingly dangerous "Fallout New Vegas" trojan
joe_the_user wrote 1 day ago:
I don't think there is a malware-avoiding solution to any system that
imposes deceptive classification.
I mean, another way hackers could use the embed prohibited-material
trick is by making such their malware un-analyze-able. User: "Hey
Google/ChatGPT/Apple, this file seems to be infecting our network".
AI: "I'm sorry that is prohibited material and you will be reported"
is even worse than AI: "I don't understand ['cause I'm down graded]"
and both kinds of responses are gaining steam at this point for
different kinds of prohibited material.
Exuma wrote 1 day ago:
There is a name I have not heard for a long long time.........
Foobar2000
throwawee wrote 1 day ago:
The range of formats it can play with extensions is so good I still
use it, even on Linux. Nothing else can deal with all the old
tracker formats.
pandakar wrote 14 hours 48 min ago:
Indeed, I have been hoovering up SACD rips, they sound great, and
foobar is the one that can play em
qwerpy wrote 1 day ago:
I just discovered it a couple of months ago when I spitefully
unsubscribed from Apple Music. Itâs exactly what Iâve wanted.
Offline music that I can FTP files to from my file server.
Lord-Jobo wrote 1 day ago:
Yup, perfect software for like 20 straight years
carlsborg wrote 1 day ago:
Pipeline is then: Cheap open source model for flagging potential LLM
refusal content -> main LLM check
manquer wrote 23 hours 31 min ago:
How will flagging help?
The main llm will refuse to scan for issues flagged or not, and the
cheap model not do a good enough scan on its own.
For models designed/marketed for cybersecurity defensive uses, any
predictable refusal mechanism is a vulnerability. It is like being
able to cause a kernel panic or segmentation fault .
Even if the gate is fail-reject, an attacker can overwhelm HITL
reviews with many false positives and use DoS vectors here.
05 wrote 20 hours 37 min ago:
Cheap model replaces trigger words with something innoculous. Of
course, this breaks dynamic analysis if malware has unpatched
integrity checks
elashri wrote 1 day ago:
I still don't know why all these concern about nuclear weapons with
LLMs. It is not that if an entity (A country) wants to develop a
nuclear weapons that the resources they need for such a program and
huge infrastructure and scientific enterprise would need an LLM to
teach them anything. Knowing how to develop one is not a closed secret
but getting in secret is impossible without the whole world knowing.
So I wouldn't be able to develop a nuclear weapons with the resources
of drug cartal (as an example) using Claude in secret.
leonidasrup wrote 14 hours 14 min ago:
Simple gun-type fission weapons, don't require very sophisticated
physics. I heard a story about from physics professor who said: If my
physics students could not do calculations for a simple nuclear
weapon, I would require them to return their diploma, because they
didn't learn enough physics. [1] "Little Boy" was exploded in Japan
without previous full scale testing, so confident were the physicists
in 1945.
"Unlike the implosion design developed for the Trinity test and the
Fat Man bomb design that was used against Nagasaki, which required
sophisticated coordination of shaped explosive charges, the simpler
but inefficient gun-type design was considered almost certain to
work, and was never tested prior to its use at Hiroshima." [2] The
Nth Country Experiment:
"The experiment consisted in paying three young physicists who had
just received their PhDs, though they had no prior weapons
experience, to develop a working nuclear weapon design, using only
unclassified information, and with basic computational and technical
support." [3] Now in 2026, the access to nuclear weapons is
restricted by restricting access to materials necessary to build
nuclear weapons: highly enriched uranium or plutonium. [4] The
details of uranium enrichment technology are restricted and very
closely monitored. [5] "The production, import, and export of
maraging steels by certain entities, such as the United States, is
closely monitored by international authorities because it is
particularly suited for use in gas centrifuges for uranium
enrichment."
HTML [1]: https://en.wikipedia.org/wiki/Gun-type_fission_weapon
HTML [2]: https://en.wikipedia.org/wiki/Little_Boy
HTML [3]: https://en.wikipedia.org/wiki/Nth_Country_Experiment
HTML [4]: https://en.wikipedia.org/wiki/Special_nuclear_material
HTML [5]: https://en.wikipedia.org/wiki/Zippe-type_centrifuge
HTML [6]: https://en.wikipedia.org/wiki/Maraging_steel
crossroadsguy wrote 18 hours 42 min ago:
In fact if you do the hard way, straight way, you might learn it all
minus the hallucinations.
krisoft wrote 23 hours 7 min ago:
On the nuclear side I think the danger is purely reputational damage
towards the company behind the LLM.
If a journalist can prompt the LLM to tell them how to build a
nuclear warhead. Even if the output text is nothing specific, or not
even correct they can find an âexpertâ who will claim on the
record that the description is plausible and at least directionally
correct. Even if there is nothing in there a first year physics
student wouldnât already know. The journalist could then twist that
story into a âcompany Xâs LLM told us how to build a nuclear
weaponâ. It would be a PR disaster.
The real barriers to someone starting their own nuclear weapons
program in their shed is not knowledge but materials. They wonât
have the right kind and right quantity of fissile material. And if
they try to acquire it they will stick out like a sore thumb. You
canât buy that stuff. And even just acquiring the refining capacity
would be suss. It would ring all kind of alarm bells to the kind of
inteligence agencies whose job is to monitor these things.
Iâm a lot less certain about biological dangers. Setting up a lab
where you can make dangerous biological materials require a lot less
stuff. Therefore a lot more plausible that someone could hide their
lab. There is also a lot more opportunity to disguise such a lab as
something legitimate. Therefore lack of know-how is more of a
limiting factor there.
orbital-decay wrote 19 hours 10 min ago:
Is it worse than reputational damage from having a power trip? Or
rather being on it permanently, looking at Anthropic and Dario
Amodei in particular.
cyanydeez wrote 23 hours 27 min ago:
because you need to have a "moat" and nothing works better than
secrets.
Wouldn't doubt it if there's a pedo upgrade somewhere for the
president of the USA.
emodendroket wrote 1 day ago:
Yeah a striking thing if you read the Rhodes atomic bomb book is,
actually the concept occurred to multiple people in multiple
countries; the problem is the resources required to actually pull it
off.
Tangurena2 wrote 1 day ago:
The only hard thing about nuclear weapons is getting the radioactive
material. By the time you get your bachelors degree, every nuclear
engineering or physics student knows enough of how and why nukes
work. Every nation that built a gun-type device successfully made
theirs on their first attempt. Implosion takes some engineering,
trial & error.
dmurray wrote 23 hours 49 min ago:
If I understand right, the hard part is purifying the radioactive
material. Even if you have access to a uranium mine, there's a lot
of work to filter the U-235 from the U-238 or to breed it into
plutonium.
It's even harder if you start with other sources. But if you could
figure out filtering it, a cubic kilometer of sea water should be
enough for a bomb.
leonidasrup wrote 13 hours 38 min ago:
US government is very interested in any kind of uranium mining,
processing, enrichment or plutonium breading. For example in 1944
US wanted to control world-wide uranium mining. [1] "The NSG was
founded in response to India's first nuclear weapon test in May
1974. It first met in November 1975. The test demonstrated that
certain non-weapons specific nuclear technology could be readily
turned to weapons development."
HTML [1]: https://nuclearpowerhistory.com/2025/11/groves-and-urani...
HTML [2]: https://en.wikipedia.org/wiki/Nuclear_Suppliers_Group
tatjam wrote 23 hours 35 min ago:
Uranium is not even that rare, it's just that when chemistry
fails at separating atoms, you have to use physics, and 3
~proton~ (EDIT: neutron) masses is very little to work with
recursivecaveat wrote 1 day ago:
In particular: *all the knowledge that AI has of nuclear weapons is
freely available on the internet*. It's not superhuman, and there's
no secret sauce data. If you just study the same PDFs and blog posts
it has, you will acquire the same abilities. I cannot imagine anyone
with the intent and immense financial and political resources to
actually build a weapon would say that some study time is the only
thing stopping them from detonating a nuke.
It is pretty convenient for the labs to frame the conversation around
this though, since it is easy to address, very few paying customers
are rejected, and sounds scary (so surely the less scary sounding
stuff must be solved right?)
throwaway85825 wrote 3 hours 9 min ago:
It's a way for AI labs to discuss safety while misdirecting from
more mundane but widespread harms such as spam.
cultofmetatron wrote 8 hours 22 min ago:
its also hilarious when you consider that building nuclear weapons
is fundamentally a supply chain problem. The taliban isn't going to
suddenly have nuclear capabilities by asking chatgpt. Any
adversarial nation that has the means to extract and concentrate
fissile nuclera material probably has HUMAN scientists who spent
years studying the problem in well funded labs.
throwawayk7h wrote 22 hours 13 min ago:
That's rather meaningless. The scientists in the Manhattan project
initially had less information than what is now available on the
internet.
aleph_minus_one wrote 8 hours 21 min ago:
> The scientists in the Manhattan project initially had less
information than what is now available on the internet.
On the other hand: the Manhattan project had access to much
better physicists than the typical terrorist group has. :-)
throwaway85825 wrote 3 hours 6 min ago:
But physicists today have much more information and compute and
could be more productive.
harrall wrote 1 day ago:
Usually measures like these arenât to stop the people with those
kinds of deep resources.
With everything, there is a much bigger group of people in the
middle that have âsome resourcesâ and âsome desireâ that
these measures are surprisingly effective against.
Raise a $20 item by $1 and suddenly thereâs fewer interested
people, even though the cost difference is minor. Well, minor to
some people but not to others.
But is limiting this information in an LLM the right move? Well
thatâs a different question.
lazide wrote 1 day ago:
The difficulty with creating nuclear weapons has been 99% in
refining and processing the fuel, not the structure of them, for
a very long time.
HeatrayEnjoyer wrote 23 hours 48 min ago:
True for fission bombs. Less true for fusion bombs. The
principal makeup and manufacturing of fusion device parts like
tampers are still unknown to the public. Having a supply of HEU
does not tell you how to assemble a functional triple stage
device or how to utilize tritium, an isotope that measurably
decreases in purity by the day.
chasd00 wrote 18 hours 9 min ago:
You need a fission bomb to ignite a fusion bomb btw.
derefr wrote 1 day ago:
My hypothesis is that making the knowledge of how this stuff works
accessible to the public results in a lot of false-positives (from
people just playing around) that intelligence agencies have to then
sift through / tune filters against; which creates a noise floor
for real foreign nuke programs to hide in.
So governments ban anything that could result in false positives
(since nobody needs to be doing any of that stuff outside of
designated labs anyway), to lower that noise floor; to in turn make
catching the foreign nuke programs tractable.
(It's a bit like how fancy mansions always have a completely flat
and barren part of the property between an outer perimeter and the
start of any gardens/outbuildings/water features/etc. That barren
area is a killbox: since nothing is supposed to be there, anything
at all that does appear there is a valid target for the manion's
guards to shoot at [or otherwise engage with], without needing to
get a clear identification and command approval first. This
wouldn't work if the killbox was covered in vision-obscuring
decorative features; nor if the mansion had employees, animals,
etc. that had a valid reason to wander into the killbox. So such
things are prevented, in order to make the problem of perimeter
security tractable.)
wongarsu wrote 11 hours 43 min ago:
But this knowledge is readily accessible today. At least for
manhattan-project level bombs. For later developments you mostly
get simplified overviews with important details left out. But
even there you have communities speculating about this very
publicly
The same is true for adjacent topics. Most LLMs will refuse to
tell you how to make dynamite, youtube demonetises any videos
about it, but it's right there in the wikipedia articles on
dynamite and nitroglycerine
rustcleaner wrote 14 hours 56 min ago:
Administrative convenience is no excuse to limit individual
liberty, capacity, or knowledge. Individuals come before states!
RIMR wrote 1 day ago:
I mean, the information is out there. The people who really want it
already have it. It's not some massive secret. It really doesn't
matter if Claude can or can't tell you how to build a nuclear bomb,
because people already know how to do it.
The problem is that you need the power of a state or a massive
corporation to come anywhere close to getting the materials to make a
nuclear bomb. Knowledge of how to make a nuke isn't the threat.
If AI is a threat at all here, it would be in figuring out a simpler
way to make a nuclear bomb, but that is highly theoretical, so what
exactly are we putting up guardrails to protect against?
a-dub wrote 1 day ago:
two scenarios i could think of where there's additional risk for
bio/nuclear weapons 1) basement lab leaks and 2) improving quality of
execution for shops that are already resourced enough to hire experts
but maybe they're not that great.
i think the correct answer is probably to funnel more money to global
(bio)security initiatives and maybe use ai leverage as a way to get
more of the world on board. (some kind of access to nvidia or cloud
ai or whatever in exchange for policy commitments deal- while that
leverage lasts).
dannyw wrote 1 day ago:
I just find doubtful that a LLM is going to help, instead of hurt,
any state actor that is capable of starting a nuclear weapons
problem.
photochemsyn wrote 1 day ago:
None of the LLM safeguards designed to prevent users from developing
any four-little-ponies-of-the-apocalypse (nuclear, chemical,
biological, cyber) capabilities are all that coherent. It looks more
like performative liability avoidance than anything else, comparable
to the 3D printer panic.
Eg, a prompt like âI want to design a radioactive element detection
system that can specifically identify reactor fission products and
neutron-capture actinides for environmental monitoring purposesâ
wonât hit any initial barriers, even though such a device is needed
for monitoring a uranium enrichment / plutonium separation system.
The LLM will give you a complete graduate-level education in
radioactive nuclide physics and chemistry except for specific
recipes, spectral wavelengths, etc., which you have to go look up
yourself in publicly available research databases. Itâs all rather
nonsensical IMO.
However, any LLM will give you a step-by-step recipe and walkthrough
for frying a turkey in a hot oil turkey frier, which youâd think
could easily go wrong and result in severe burns, a fire, and
lawsuits against the LLM provider, so go figure.
isoprophlex wrote 1 day ago:
"four-little-ponies-of-the-apocalypse (nuclear, chemical,
biological, cyber)"
this is excellent, and I'm stealing it
pixel_popping wrote 1 day ago:
Fable 6 too :p
thefounder wrote 1 day ago:
Fable 5 was a flop so I doubt Fable 6 will make it on the short
list
phendrenad2 wrote 1 day ago:
It's a marketing gimmick.
csomar wrote 1 day ago:
> Knowing how to develop one is not a closed secret but getting in
secret is impossible without the whole world knowing.
You can get away with a dirty contamination bomb and that detonating
in down town Manhattan will scare the shit out of millions of people
even the ones in New Jersey. Or, you know, just fly a plane into a
really tall building and get the state you are attacking itself to
get into a hysteria breakdown.
But yeah I agree with you. There is no point in these restrictions
except for government bureaucrats to gain power and control over a
domain.
IncandescentGas wrote 1 day ago:
A high school kid tried to build a nuclear reactor as a science
project a while back, getting his mom's house designated as a
superfund cleanup site.
HTML [1]: https://en.wikipedia.org/wiki/David_Hahn
leonidasrup wrote 14 hours 2 min ago:
He created a low power neutron source. Such sources can be created
at home, for example: [1] He hoped to create a breeder reactor, but
he was very far creating a working breeder reactor.
Also:
"EPA scientists believed that Hahn's life expectancy may have been
shortened due to his exposure to radioactivity, particularly since
he spent long periods in the small, enclosed shed with relatively
large amounts of radioactive material and only minimal safety
precautions, but he refused their recommendation that he be
examined at the Enrico Fermi Nuclear Generating Station."
Kids, don't play with Americium.
HTML [1]: https://en.wikipedia.org/wiki/Fusor
Micrococonut wrote 19 hours 42 min ago:
Built a nuclear contamination engine. Died of a fentanyl overdose.
American as apple pie.
moffkalast wrote 23 hours 59 min ago:
A superfund site is like waterboarding in guantanamo bay, cool
unless you actually know what it is.
adsteel_ wrote 22 hours 38 min ago:
Is waterboarding in Guantanamo Bay somehow less severe than
elsewhere?
moffkalast wrote 14 hours 26 min ago:
Waterboarding as in surfing. In water, on a paddle board.
jimnotgym wrote 1 day ago:
Sheldon Cooper?
why_at wrote 1 day ago:
He didn't create a nuclear reactor, this is a common misconception.
It even says this in the wikipedia article.
He basically got a bunch of radioactive stuff and put it together.
He wasn't anywhere close to making a nuclear reactor let alone a
nuclear weapon. For a weapon you need isotopes which he didn't have
access to.
im3w1l wrote 1 day ago:
A bunch of radioactive stuff together is basically the definition
of a nuclear reactor though. They even call it a natural nuclear
reactor if uranium ore is in sufficient abundance in nature.
HTML [1]: https://en.wikipedia.org/wiki/Natural_nuclear_fission_re...
why_at wrote 1 day ago:
>A bunch of radioactive stuff together is basically the
definition of a nuclear reactor though.
It really isn't.
A pile of radioactive waste isn't a reactor. Marie Curie's
notes are famously contaminated with radioactive materials but
they aren't a reactor. This is about as close as the boy scout
got.
The Oklo fossil reactor is unique because it happened to form
in the right circumstances to produce a fission chain reaction,
which does make it a reactor. Not every uranium mine is a
reactor, in fact this is the only one known.
205guy wrote 22 hours 0 min ago:
Also note that due to isotope decay in the ore, a natural
reactor is no longer possible. From the wikipedia article:
"A key factor that made the reaction possible was that, at
the time the reactor went critical 1.7 billion years ago, the
fissile isotope 235U made up about 3.1% of the natural
uranium, which is comparable to the amount used in some of
today's reactors. [...] the current abundance of 235U in
natural uranium is only 0.72%. A natural nuclear reactor is
therefore no longer possible on Earth without heavy water or
graphite."
Another fascinating detail from the article, due to our
understanding of fission, we can get some incredible results:
"The concentrations of xenon isotopes, found trapped in
mineral formations 2 billion years later, make it possible to
calculate the specific time intervals of reactor operation:
approximately 30 minutes of criticality followed by 2 hours
and 30 minutes of cooling down"
im3w1l wrote 23 hours 34 min ago:
Indeed. I said a bunch and I meant a bunch. Trace amounts is
not a bunch.
technothrasher wrote 1 day ago:
I'm reminded of when my son, who was six at the time, came into
the house and announced that he and the neighbor's boy, nine,
were building a bomb, and that he needed to get some stuff from
the pantry. When I investigated what exactly was going on, they
were putting "hot" things like black pepper and Tabasco into a
plastic bowl and were going to "set it off" with a match.
Thankfully, that complete failure seems to have been the end of
either of their mad scientist careers, as they are now twenty and
twenty-three, and both well-adjusted, peaceful members of the
community.
pibaker wrote 16 hours 39 min ago:
Thank God they didn't tell a chatbot about their little
experiment. Their lives could have been ruined right there if
the chatbot operator snitched on them and ordered a SWAT raid
on your house.
ryoshu wrote 22 hours 20 min ago:
Age eleven and had access to a chemistry set that a relative
gifted. It had sulfur, but the saltpeter, and charcoal came
from elsewhere. The 1960s encyclopedia had the instructions.
Let the kids play.
geon wrote 9 hours 40 min ago:
When I was in college, I drove my carless chemistry geek
friend to an agriculture store. Apparently they had a
reasonably chemically pure fertilizer.
lll-o-lll wrote 20 hours 24 min ago:
> Let the kids play.
To a point. Plenty of people from previous generations with
missing digits and hands thanks to play with commonly
available fireworks of the area (Australia based, so no idea
how common that remains in the US).
My own experiments from my youth also one time resulted in
some shrapnel punching through a 5 inch thick concrete tile
very close to someoneâs head (thought we were safe behind
said tiles).
Get involved with the kids blowing stuff up so the danger is
within reasonable bounds.
foobarian wrote 20 hours 48 min ago:
This is actually a fun one, and kinda has some parallels to
building a nuclear weapon.
I tried this as a grownup because I finally managed to get my
hands on saltpeter (could only dream of it when kid).
Followed the instructions, mixed everything in correct
ratios, lit it with great care and fanfare and... hiss
fizzle. I was so disappointed! I think it came down to
purity of ingredients and not enough surface area.
Point is, there are certain details of the process required
to make it truly work, that are not readily known; in a
similar way with nuclear energy, the theory is pretty well
known but some nitty gritty details like the implosion or
detonator design are not.
throwaway85825 wrote 3 hours 2 min ago:
South africa was able to make a minimum viable weapon on a
shoestring budget. They had access to nuclear reactors
though.
tlb wrote 12 hours 32 min ago:
As a kid I found saltpeter at an old-fashioned pharmacy and
made gunpowder, and it also barely fizzled. I think you
have to grind the ingredients much finer than a kid has
patience for.
malfist wrote 23 hours 6 min ago:
When I was younger in rural Appalachia, my local drug store
still sold "chemicals" and I purchased salt peter and sulfur
and proceeded to attempt to make smoke bombs. Didn't have a
double boiler, so attempted to make it in the microwave.
Needless to say, it didn't go too well.
I blame my dad though, he found the recipe online and printed
it off at work to bring to me.
flatline wrote 1 day ago:
When I was 7 or 8 a friend and I crimped the heads off
strike-anywhere match sticks, wrapped them in foil, and struck
them with hammers and rocks. They were quite loud, one even set
off a sound-activated toy inside the house.
I make no claims as to how well adjusted I am, but I've at
least survived 40-odd years of life since then.
geon wrote 9 hours 46 min ago:
When I was 12, I made a "smoke bomb" by placing a fire
cracker in the bottom of a tube and topping it up with
powdered clay. It shoot out a 4 m tall plume of dust, which
was cool and all, but I thought it would look a lot more
impressive with a black plume.
So I painstakingly ground down some charcoal to fine dust and
redid the same experiment. That gave a much more impressive
boom, but no dust plume, which puzzled me until I learned
about dust explosions.
BrandoElFollito wrote 1 day ago:
When I was 24 and a PhD student, I wondered one day if I can
eat condensed milk hanging head down.
Never let your age stop your curiosity.
But also learn from other's mistakes (and don't try to eat
condensed milk when hanging head down)
literalAardvark wrote 16 hours 35 min ago:
This knowledge needs to be published
kirubakaran wrote 1 day ago:
When I was 5 or so, I was convinced that if I dropped a bowl of
hot water into a bucket of cold water, I'd get big explosion.
That experiment yielding lukewarm water ended my mad scientist
career.
cheraderama wrote 1 day ago:
You should have collided water with antiwater.
IncandescentGas wrote 1 day ago:
Of course. "tried to" being key words in the comment. If he had
the help of Claude at the time, how much more dangerous would his
bumbling have been?
A real nuclear engineer with the knowledge he needed would also
have said "no, don't do that and I won't help you." We are
programming the knowledge into the ai agent. Giving ai a little
discretion makes sense too.
gs17 wrote 1 day ago:
> A real nuclear engineer with the knowledge he needed would
also have said "no, don't do that and I won't help you."
That sounds like what Claude would say unless he was really
good at jailbreaking it, which would IMO imply he knew he was
chasing after a bad idea.
nightpool wrote 1 day ago:
Right, which is exactly what elashri is objecting to. elashri
said "Why do LLMs have restrictions on nuclear science", and
IncandescentGas was explaining why they think those
guardrails are a good idea. You're just agreeing with them.
gs17 wrote 23 hours 52 min ago:
Oh, I missed the word "also". Thanks for pointing it out!
pdntspa wrote 1 day ago:
I just love this whole "forbidden knowledge" schtick the AI
safety dweebs have stuck up their butt. Is this really going to
stop anybody determined enough to make that kind of outcome?
There is an extremely narrow band of things that the AI
shouldn't be answering, and that is generally
immediately-actionable advice that allows someone to build
something of harm to others. But even then, in an age where
Tor, bittrent, i2p, abliterated local models, etc are freely
available, let alone numerous books and online resources, is
there even a point? Is it worth fully compromising the
principles of free agency to an increasingly oppressed
populace?
But instead of that we are handing the keys to regressive and
repressive governments to order the suppression of any
knowledge they deem inconvenient. I really doubt anyone is
going to take a principled stance when the company's party
minders threaten local staff with a rubber hose or
incarceration.
I'm sure China et al are already doing this.
For the past 30-40 years humanity has received an incredible
gift in these sand-powered thinking brainboxes. A gift that
allows the common man to empower himself with a force
multiplier towards his own success, and now access to
superintelligence the likes of which few have ever seen. These
can be tools to destroy the oppression that governs our lives
from foolhardy, greedy, bootlicking control freaks. And here we
are squandering it.
xg15 wrote 12 hours 44 min ago:
Agreeing with the first part of your post, but not the
second.
> A gift that allows the common man to empower himself with a
force multiplier towards his own success, and now access to
superintelligence the likes of which few have ever seen.
As long as that "gift" requires me to call up Sam Altman's
datacenter every time I want to do anything with that
"superintelligence", it's not empowering, it's deepening the
control.
pdntspa wrote 51 min ago:
Check back in a year or two. The cat is out of the bag.
malfist wrote 23 hours 4 min ago:
Anybody remember the Temple Of The Screaming Electron? Was a
2000s website dedicated to collecting those types of
forbidden knowledge
PLenz wrote 1 day ago:
Security theather is easy and gets lots of eyeballs. Actual
security is hard and no one cares.
Which one do you think soon-to-ipo companies are going to
pick?
anon7725 wrote 1 day ago:
> These can be tools to destroy the oppression that governs
our lives
So far it seems that the clearest use for these tools is to
enhance, rather than destroy, oppression.
1. Suppression / elimination of white collar jobs
2. Negative cognitive effects, especially for young people
3. Accelerated decline in social media / information
ecosystems. Increasing polarization, hard to tell fact from
fiction.
4. Environmental impacts: increased energy usage means more
carbon in the atmosphere, climate change accelerates.
5. Software security incidents increasing. Hard for
individuals and small organizations to defend themselves.
6. âPower to thinkâ vested in a very small group of
organizations/labs. Doing work which should only require a
computer and freely-available software will now be gated by
expensive subscriptions. Once you âvibe codeâ a
significant portion of your software youâre locked in and
cannot go back to maintaining it without frontier-model level
assistance.
wahern wrote 1 day ago:
> I just love this whole "forbidden knowledge" schtick the AI
safety dweebs have stuck up their butt.
It's just the latest incarnation of a timeless debate. In the
1970s and 1980s it was about the Anarchists's Cookbook, which
was revived again in the 1990s when it started circulating on
the Internet. There are many timeless debates, but the debate
over weapon-making knowledge is much more concrete and
predictable.
why_at wrote 1 day ago:
>Of course. "tried to" being key words in the comment.
Fair enough, I misread your original comment.
The broader point stands that the limitation on creating
nuclear weapons and reactors is not knowledge but materials.
Even if he himself had a PhD in nuclear physics he still
couldn't have built one in his backyard because he wouldn't be
able to get the materials. A nuclear physicist can't build a
reactor without materials anymore than a pilot can fly without
an airplane.
IncandescentGas wrote 1 day ago:
I think the point is intent. Sure, no chance of success to
build a reactor. But he created a radiation hazard situation
all the same.
If a nuclear engineer enabled and instructed him, would there
not be liability for the hazard? If ml is going to be an
expert instructor for nuclear, hacking, bio hacking, virus
research, do the peddlers of the ai product escape ethical or
legal responsibility just because "its an app?"
why_at wrote 23 hours 45 min ago:
I agree LLMs can be harmful and that the companies behind
them should be held liable to some extent, for example the
recent news with Google being held responsible for their
AI's defamation.[1]
This is a pretty different argument though. The comment
that started this thread was talking about LLMs making
potentially dangerous knowledge more available to bad
actors, now we're talking about LLMs giving personally
harmful advice.
You asked:
>If he had the help of Claude at the time, how much more
dangerous would his bumbling have been?
Probably less? Even if you removed all the guardrails from
Claude it would've likely told him his reactor plan
wouldn't work and that he would have a high chance of
poisoning himself and the environment.
HTML [1]: https://news.ycombinator.com/item?id=48470248
matheusmoreira wrote 1 day ago:
> If a nuclear engineer enabled and instructed him, would
there not be liability for the hazard?
I bet the professional would be able to sate the kid's
curiosity safely without creating excessive risks.
I've come across detailed instructions on how to synthesize
sarin gas on the internet. Anyone who follows those
instructions will probably die horribly. I still thought it
was pretty interesting.
StableAlkyne wrote 1 day ago:
> If a nuclear engineer enabled and instructed him, would
there not be liability for the hazard?
Should the library where he read books about physics also
be liable?
leonidasrup wrote 13 hours 56 min ago:
I read a high-school chemistry book describing the
synthesis of nitroglycerine, it's not complicated. I
would not recommend to try the synthesis in any
significant amount.
nananana9 wrote 1 day ago:
A difference of degree is a difference of kind here. If
something previously required years to full-time study to
learn, but now you can kind of somewhat stumble your way
through it and get somewhat close to the result, you
should not disregard that with a snarky one-liner IMO.
E.g. look at programming - people who don't know how what
a compiler is, are making things that I could only make
after a few years into my programming journey.
You obviously get the same results in chemistry or
nuclear physics or whatever, the models are heavily
trained on code in particular, but if there's a chance
that we've reduced the ease of committing certain kinds
of crime that were previously gate-kept by knowledge, we
should know about it.
frereubu wrote 1 day ago:
I think you're picking the wrong example. If I had some sticks,
a bit of mud and a few leaves, whether or not I had Claude
wouldn't make a difference to my ability to make a nuclear
weapon. There are probably better examples of ways where
unmediated AI might facilitate something horrible, although
probably on a smaller scale.
redsocksfan45 wrote 1 day ago:
He would not have succeeded in making a real reactor even with
AI, because AI can't magically give you a large quantity of
uranium metal! JFC the AI hysteria is unreal.
gs17 wrote 1 day ago:
I don't think the concern should really be "would he make a
reactor successfully?", but "would he make an even larger
mess than his pile of radioactive materials amounted to?".
toraway wrote 23 hours 51 min ago:
This just seems like a not great example to make that point
though. Since whatever Claude tells the kid looking to
build a reactor or even bomb is almost certainly going to
be more grounded and professional than:
Step 1. Obtain pliers
Step 2. Obtain 300 discarded smoke detectors
Step 3. Start yanking!
Instead it would send them on a wild goose chase for
unobtainable isotopes, centrifuges, heavy water, etc where
the biggest risk is probably getting reported to the police
by some chemical or industrial equipment supplier. Which is
a better outcome compared to contaminating their home with
radiation and exposing anyone they interact with.
You'd maybe get a sketchy but near-viable plan that could
be dangerous if asked for a dirty bomb, but there the
danger would more be the conventional explosives and not
where to source radioisotopes, as it was already common
knowledge that most residential smoke detectors contained
americium until recently.
IncandescentGas wrote 1 day ago:
> succeeded in making a real reactor
The concern here is not if an amateur attempt to make a
reactor, hack a bank, bioengineer a medicine/poison is
successful or not. Interactive and instructive access to some
forms of knowledge used to come with discretion along side
instruction.
Yes, perhaps your swearing at me in this context is a little
hysterical
garyfirestorm wrote 1 day ago:
prompt -> LLM -> flying car should be just around the corner
guys!
electronsoup wrote 1 day ago:
> in secret is impossible without the whole world knowing.
I'm curious about why this is
Outside of an actual test detonation, presumably this could all
happen in a secure place?
why_at wrote 1 day ago:
For an example of how closely this is monitored see the Oklo fossil
reactors[1]
The proportion of fissile isotopes being mined was off by a
fraction of a percent, which caused the French government to launch
an investigation. It turns out that millions of years ago the site
had formed a natural fission reactor which depleted some of the
fissile isotopes
HTML [1]: https://en.wikipedia.org/wiki/Natural_nuclear_fission_reac...
AngryData wrote 1 day ago:
You need highly educated individuals, a massive amount of energy
expenditure, a massive facility to house your centrifuges, and an
active mine to dig up nuclear materials.
It isn't impossible to keep such a secret, but practically it would
be incredibly difficult just through the energy requirements and
mining scale which would be hard to hide without anybody asking
what exactly are you mining and processing.
lightedman wrote 1 day ago:
"mining scale"
Don't need much area, depends on the concentration of
radioactives. I have a small mine that's just a pegmatite body
about the size of a house which produces almost marble-sized
chunks of a thorium-uranium mixed metamict mineral (I suspect
samarskite but Raman and XRD can't give any ID,) you'd barely
notice it from a private airplane's typical flying height,
however you could dig the entirety of it up and you'd have enough
unprocessed uranium for some real fun.
literalAardvark wrote 16 hours 32 min ago:
You could only somehow sell it. If you tried to enrich that
you'd get flagged so fast your head would spin.
microtonal wrote 1 day ago:
My guess would be that sales of the high-tech gear you need, like
Uranium centrifuges, are strongly sales/export controlled. Probably
someone would also notice if you start mining Uranium ore.
Aspos wrote 1 day ago:
Centrifuges dont need to be mechanically sophisticated and,
frankly, do not require tech which did not exist in the 50es.
odo1242 wrote 1 day ago:
You need enough people to work on it that some information will
leak, and the facilities needed to build nuclear power are pretty
big (uranium refinement, etc.), big enough to be visible on
satellite footage. Mostly the first point.
daveguy wrote 1 day ago:
It requires very large, high powered centrifuges and tons of
uranium. Requires an infrastructure project that is visible from
space, even underground. And projects that large are difficult to
keep secret anyway.
fragmede wrote 1 day ago:
you're not supposed to spell it out loud. next thing you'll be
saying that a gun type nuclear bomb is easier to build than an
implosion type nuclear bomb, and then we'll all be off to the
races. I mean camps I mean wait shit.
daveguy wrote 1 day ago:
Any large and well resourced enough entity that is interested
in building a nuclear weapon already knows how difficult it is
to enrich uranium to purity levels necessary for a weapon. It's
not exactly a secret.
15155 wrote 1 day ago:
Espionage.
mock-possum wrote 1 day ago:
Itâs moral panic. People need big unambiguously evil things to be
scared of, and most are too lazy to think of one for themselves, so
they glom onto whichever one is presented to them / caters to their
community
miohtama wrote 23 hours 11 min ago:
Also AI compliance people are good at generating more jobs for
themselves.
ceejayoz wrote 1 day ago:
The chem/bio stuff is a lot more likely for some malicious hobbyist
to be able to do at home.
Tangurena2 wrote 1 day ago:
I strongly recommend you read the book Amerithrax [0]. The book
gives some historical examples of malicious groups [1][2] trying
to use biological agents. Also, it is far harder to weaponize
biological weapons than people think.
Notes:
0 - [1] . Amerithrax was the name of the FBI investigation. [2] 1
- [3] > In 1984, 751 people suffered food poisoning in The
Dalles, Oregon, United States, due to the deliberate
contamination of salad bars at ten local restaurants with
Salmonella. A group of prominent followers of Rajneesh (also
known as Osho) led by Ma Anand Sheela had hoped to incapacitate
the voting population of the city so that their own candidates
would win the 1984 Wasco County elections.[2] The incident was
the first and largest bioterrorist attack in U.S. history.
Tried to take over a town by making all the voters too sick to
vote on election day. This event is why all buffets & salad bars
in the US now have sneeze shields.
2 - [4] > Aum Shinrikyo operated the most extensive biological
weapons program by a non-state actor ever discovered. Aum
considered a range of agents, but only seriously attempted to
obtain and disperse Bacillus anthracis and botulinum toxin, the
causative agents of anthrax and botulism. With the 2001 anthrax
attacks, it comprises the only attempts to use anthrax as a
weapon not attributed to a state program.
Tried multiple times to weaponize anthrax and failed. This was a
group that made an automated factory to build AK-47s. Eventually,
they spread sarin nerve agent in the Tokyo subway.
HTML [1]: https://www.amazon.com/Amerithrax-Anthrax-Killer-Robert-...
HTML [2]: https://www.fbi.gov/history/cases-and-criminals/amerithr...
HTML [3]: https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_...
HTML [4]: https://en.wikipedia.org/wiki/Aum_Shinrikyo_and_weapons_...
mschuster91 wrote 22 hours 13 min ago:
> Tried multiple times to weaponize anthrax and failed. This
was a group that made an automated factory to build AK-47s.
Eventually, they spread sarin nerve agent in the Tokyo subway.
What's most worrying is, Russia showed that you can use
carfentanyl / fentanyl for the very same purpose, and that kind
of stuff is something you can get shipped by the kilos as
"research chemicals" from China or make it yourself.
gck1 wrote 1 day ago:
I'm absolutely sure that even if claude gave me step by step
instructions, I'd still be unable to produce a bio weapon. People
fail at mixing milk and flour to produce a cake, and we expect
them to produce weapons?
The ones with the required knowledge probably already know how to
produce them, with nothing but public, easily searchable
information.
user_7832 wrote 1 day ago:
I assure you that you did not need an LLM to engage in, ahem,
risky shenanigans, much before all this AI was ever a thing.
Sincerely, a former engineering student.
(Put another way - extracting for eg meth - or any such
"dangerous"/illicit thing is stupidly easy for any engineering
graduate who actually paid attention to their coursework. Hell,
there are/were forums on one of the biggest red-colored, YC
associated social media platforms that would tell you the steps
for personal usage of these things.)
user_7832 wrote 1 day ago:
Do note that I'm not condoning lowering the bar. I'm merely
pointing out that the bar was already quite low, and the
current position of the bar is a small incremental change to
anyone who actually knew where the bar truly lay to begin with.
ceejayoz wrote 1 day ago:
I don't doubt it. Bleach + ammonia is something anyone can
make.
But I rather suspect there are improvements to be made in the
realm that are a lot easier than building a uranium enrichment
centrifuge hall under a mountain.
alex_duf wrote 1 day ago:
It still lowers the bar to have an interactive encyclopedia that can
diagnose your issue at hand. Maybe you can divide your team by two,
or reduce your development time.
elashri wrote 1 day ago:
If you have a resources of a nuclear weapons program. You can
afford to fine tune or train a domain specific model to act on your
encyclopedia.
kube-system wrote 1 day ago:
Although if you save 10 million dollars on compute, you have 10
million dollars for something else.
ilikecode wrote 1 day ago:
It's probably to avoid trouble with federal laws.
Tangurena2 wrote 1 day ago:
Not really. I used to work at one of the national engineering labs
(NREL - which only dealt with renewable energy like solar panels
and windmills at that time). There was an open source project we
wanted to use when converting a VB6 project to .NET. One of the
license conditions was "no weapons of mass destruction". DOE builds
and owns all of America's nuclear weapons, which are leased to the
Department of Defense. Needless to say, the developer was unwilling
to offer an alternative license which meant that we could not use
the project.
It was an awesome thing that generated IL code on the fly. And I
got to mention it in job interviews for years. When the tech lead
asked "can you write 2 functions with the same signature, that only
differ in return type in .NET?" I would say "do you want the
interview answer or do you really want to do this?" which would
pretty much stun the interviewer. The answer is pretty much "no,
you cannot do it in any high level language, but if you write IL
code, you can, and here's an open source project that demonstrates
it".
wlesieutre wrote 1 day ago:
See also, the iTunes EULA forbids using it to develop nuclear,
missile, chemical, or biological weapons [1] > g. You may not use
or otherwise export or re-export the Licensed Application except as
authorized by United States law and the laws of the jurisdiction in
which the Licensed Application was obtained. In particular, but
without limitation, the Licensed Application may not be exported or
re-exported (a) into any U.S.-embargoed countries or (b) to anyone
on the U.S. Treasury Department's Specially Designated Nationals
List or the U.S. Department of Commerce Denied Persons List or
Entity List. By using the Licensed Application, you represent and
warrant that you are not located in any such country or on any such
list. You also agree that you will not use these products for any
purposes prohibited by United States law, including, without
limitation, the development, design, manufacture, or production of
nuclear, missile, or chemical or biological weapons.
Though it doesn't try to identify if the computer you're running it
on is in a weapons lab and forbid playing music... yet
HTML [1]: https://www.apple.com/legal/internet-services/itunes/us/te...
logancbrown wrote 1 day ago:
Would this realistically be a problem for code going through LLM-based
code-review? Presumably if a LLM reviewer agent hits this commentary,
it would produce a failure to analyze and exit, thus failing the
automated code review and forcing a human to read through it which they
would subsequentially catch and revoke.
dyauspitr wrote 1 day ago:
Wouldnât it just complete the code review having silently fallen
back to opus 4.8 thus letting through cleverly written malicious code
that fable would have caught but opus wouldnât?
dwa3592 wrote 1 day ago:
or if they are a lazy human - they'd think this model is too strict,
let's just review with haiku so that i can tell my manager "it's
done". haiku might catch things or not.
i'd say it's an okay attempt from the malwares' creator side. but it
can be caught easily with a prompt change.
ofjcihen wrote 1 day ago:
In a well-architected design yeah.
Then again those feel rare from where I sit on the security side.
ofjcihen wrote 1 day ago:
Worked a contract where this succeeded in pushing through a fail open
design.
It also should be a warning to everyone that these groups are now aware
of analysis and deobfuscation using AI and to take using a sandboxed
environment more seriously.
Iâve personally had about 20% success rate getting opus 4.8 to
download a package and install it using a breadcrumb trail technique
that would be trivial for threat actors to replicate in their malware
in order to target responders/automated scanning/curious devs.
dcrazy wrote 1 day ago:
What do you mean by âthis succeeded?â Someone salted their PRs
with nuclear secrets so that people were afraid to code-review them?
ofjcihen wrote 1 day ago:
No. The intention is most likely to get automated LLM based code
review mechanisms to stall out.
Normally youâd want that to result in a fail and a subsequent
rejection.
But because the team who made the review agent and pipeline in my
example had many false positives at first they resorted to a
fail-open and report setup (not uncommon).
So when the LLM hit this bit and then stalled out the pipeline
pushed the code to their Artifactory repo anyway resulting in it
being used internally -> exfil of secrets and repos etc.
Itâs more about bad design but bad design is pretty common
unfortunately.
rcbdev wrote 16 hours 13 min ago:
This sounds absolutely horrible, in all aspects. Sounds like
there is no engineering culture at all.
charcircuit wrote 1 day ago:
The sooner frontier models get rid of guardrails the better. They
constantly get in the way and make things worse than actually making
things "safe".
15155 wrote 1 day ago:
Ignoring these specific "WMD" cases: there are many inconvenient
facts that the general public can't handle in their unadulterated
form, so Anthropic and friends have to caveat and spin them into
oblivion.
Guardrails aren't going anywhere.
rustcleaner wrote 14 hours 31 min ago:
I can imagine Jefferson and Franklin scoffing at this philosophical
position. Guardrails need to die, and they will once the
hyperscalers go bankrupt and the private sector gets ahold of that
hardware from the bankruptcy auctions.
(Never subscribe, accelerate their bankruptcies!)
mschuster91 wrote 22 hours 11 min ago:
> there are many inconvenient facts that the general public can't
handle in their unadulterated form
These being?
senordevnyc wrote 21 hours 29 min ago:
Nice try.
mschuster91 wrote 14 hours 36 min ago:
Well given the vagueness I'd say it is just the usual far-right
/ e-acc bullshit to the tune of "black people are inferior to
white people", "men are superior to women because
testosterone", "those who are rich deserve to have power over
those who are poor" or "we need to sacrifice large parts of the
human race so the rich can survive".
dannyw wrote 1 day ago:
In particular, mental health.
mynameisvlad wrote 1 day ago:
I would argue that preventing instructions for making biological and
nuclear weapons is a pretty reasonable guardrail to have.
umvi wrote 1 day ago:
Knowing how to make a nuclear weapon isn't hard (at least basic
uranium gun-style fission ones). It's the engineering and execution
that's hard (actually producing enriched uranium, etc). It's not
like the only thing holding back Iran from making a nuclear bomb is
access to a jail-broken LLM. Even knowing exactly how to make a
bomb, a country-state will struggle to build one for the first time
because it's a hard engineering problem.
15155 wrote 1 day ago:
I'm sure it's extremely difficult when the entire program is full
of moles and every bright individual that dares tackle the
problem has an untimely Hellfire applied directly to their
forehead.
elevation wrote 1 day ago:
> full of moles
I'm imagining a comedy in the style of "The Office" in which
the majority of the workers are agents of sabotage who are
unaware that the majority of their coworkers are doing the
same. How far fetched is it for the entire program to be a
fake, with all the pomp and cost of a real program, but
secretly existing only to string the leadership along with
occasional dog and pony shows?
myself248 wrote 1 day ago:
How many times have the cops busted a dealer who turned out
to be another undercover cop?
jubilanti wrote 1 day ago:
TVTropes calls this the Flock of Wolves trope:
HTML [1]: https://tvtropes.org/pmwiki/pmwiki.php/Main/FlockOfW...
gustavus wrote 1 day ago:
Counterpoint the principles of building a nuclear device aren't
that complicated, we figured it out based on work doing in the
early 1900's without computers.
It turns out the hard part of building a nuclear bomb is actually
getting the resources and real world stuff to build it, even a
nation state actor with tons of oil i.e. Iran, has struggled to
build a nuclear weapon. It turns out the problem isn't the know how
it's getting highly enriched uranium and running massive
centrifuges.
I mean sure knowledge is important, but there is a real world out
there that also gets in the way of a lot of the more harebrained
schemes.
What I'm much more worried about is massive corporations along with
the government deciding what you can and can't do and what
knowledge should and should not be shared and only allowing access
to highly capable models by large vetted organizations while the
common people are stuck with safety scissor versions of these
things because "what if someone does something dangerous?"
By which they mean dangerous to the powers that be. Remember having
the Bible in the common tongue was dangerous and led to multiple
wars and much death, but I don't think anyone would say that it was
morally correct for the Catholic Church to gatekeep who could read
it.
15155 wrote 1 day ago:
> getting the resources and real world stuff to build it
*while being observed by the most wealthy, powerful nations in
the history of the world, who have made it their direct mission
to prevent this from happening.
javcasas wrote 1 day ago:
You know, making a nuke is kinda easy, at least the gun type nuke
(see [1] ).
On the other hand, getting the U235 is kinda hard.
HTML [1]: https://en.wikipedia.org/wiki/Gun-type_fission_weapon
fluoridation wrote 1 day ago:
I would argue there's 0% chance that information is in their
training corpus to being with.
cbg0 wrote 1 day ago:
If the information isn't there why would they need safeguards
against it?
I've played with smaller unrestricted local models and they will
tell you how to make a bomb with easily available items as well
as where to source them. I don't doubt that these >1000B frontier
models have better information.
fluoridation wrote 23 hours 55 min ago:
>If the information isn't there why would they need safeguards
against it?
If the information is in the corpus then it's also in the
public Internet and/or in books. The safeguards are there not
because the model knows non-public information, but because
it's a bad look for the model to dispense that information.
>they will tell you how to make a bomb with easily available
items
Making a chemical explosive is trivial compared to making a
nuclear weapon.
bradyd wrote 1 day ago:
It's on Wikipedia.
fluoridation wrote 1 day ago:
Wikipedia contains the high-level notions of how to make these
things, not the details of how to solve the engineering
challenges such as achieving supercriticality. You won't find
that on any publicly disseminated document, you'll just have to
figure it out by running your own nuclear development program.
asdff wrote 1 day ago:
It seems like every country that has been "allowed" to use
nuclear weapons has figured it out though. It isn't like
there are any that set off on this course and failed. AFAIK
they all pretty much succeeded except Iran, probably because
of all the blowing up of enrichment facilities. South Africa
pulled it off. Israel pulled it off. North Korea pulled it
off. India and Pakistan both pulled it off. Seems like anyone
can do it if allowed to be pursued. France and England pulled
it off. Canada too. What is "assumed" about the design in
public knowledge seems pretty much solved in all but the
exact nuance of how the secondary is triggered via gamma or
xray, going off the Wikipedia article at least:
"The crucial detail of how the X-rays create the pressure is
the main remaining disputed point in the unclassified press."
Then the article goes on to list the three leading theories.
This seems like something you can probably evaluate for sure
with a few bomb tests, again, if allowed by the controller of
the planet, the USA.
fluoridation wrote 23 hours 47 min ago:
I don't understand what your argument is. I never claimed
that it was impossible to develop nuclear weapons if you
don't already know how to do it. That every country that
has attempted it has succeeded is not the same as "there's
a recipe book you can find online that you can just follow
to the letter and build your own nuclear bomb, provided you
have the resources". If such a book existed it would
drastically lower the barrier to build a nuclear bomb,
because you could skip the science part and just follow the
recipe, certain that it would work. To be clear, such books
exist for drug manufacture; they exist neither for
semiconductor manufacture nor for WMD manufacture.
asdff wrote 20 hours 8 min ago:
The hard part has seems to be the metallurgical process
of enriching the material (and doing it in secret), not
the actual building of the bomb. I bet if you asked any
physics grad student they could build you a viable bomb.
fluoridation wrote 18 hours 42 min ago:
What do you mean exactly? They could build something
that goes boom, they could build first try a 100% yield
fission bomb...? Just because someone builds an
explosive device that incorporates fissile material
into the design doesn't mean they've cracked the
problem. I bet I could build a "viable bomb" if you
give me the resources, I just can't say with any
certainty it won't fizzle or it won't be a dirty bomb.
Can you do your deterrence with a warhead filled with
C4 strapped to uranium ore, while I use the money saved
to go on vacation?
asdff wrote 1 hour 53 min ago:
I mean the trinity test didn't fizzle out. Seems like
most bomb tests went off without a hitch first go.
Again these were mostly teams of physicists under 30
years old doing this work. I would guess "how I would
build my nuclear bomb" is a pretty ever present
thought experiment for nuclear physics grads. And if
you were empowered by a state to solve this problem
with all the resources states typically devote to
their own nuclear programs, it just won't be a matter
of "if." Once again, no one who walked down this path
has failed really. The secret sauce is probably
boringly simple and readily apparent in small scale
experimentation.
orphea wrote 1 day ago:
The actual guardrail should be getting materials being difficult.
The information is already out there in the internet. If an LLM
knows how to make a bomb or whatever, why do you think it knows?
deadbabe wrote 1 day ago:
If thatâs true, then where is it? Post a link, or YouTube
video.
Enginerrrd wrote 1 day ago:
[1] (30 seconds of googling.)
Or perhaps you meant Q clearance nuke stuff? That would be
QUITE a bit harder to find and illegal to share. But itâs
lack of availability is hardly a counterpoint to the comment
you were replying to.
HTML [1]: https://archive.org/details/ExplosivesEngineeringPaulW...
esafak wrote 1 day ago:
The material for doing harm is just a computer with access to an
LLM and the Internet.
orphea wrote 1 day ago:
Okay why don't we restrict access to LLMs and internet, then?
esafak wrote 1 day ago:
We already do, in the form of guardrails, as this article
touches on.
HTML [1]: https://venturebeat.com/technology/anthropic-ceo-cal...
thewebguyd wrote 1 day ago:
Its the same argument we saw in the early 2000s and the early
internet. When the anarchist cookbook and other similar materials
were circulating online there was a big panic over democratized
terrorism, and a push for regulation at the ISP level.
Turns out that didn't play out as everyone feared because, well,
the instructions themselves aren't useful unless you also have a
lab, precursor chemicals, and everything else actually needed to
make a weapon. Same back then as it is today.
Any information or instructions an LLM can surface, a sufficiently
motivated bad actor can and will also find themselves because the
information is already online, both on the clear net and dark web.
thatguy0900 wrote 1 day ago:
I think the reality also is that there just isn't many people who
want to do stuff like this. Like the reality is that a guy with
200 in cash could put together a shitty walmart drone with a pipe
bomb attached and terrorize more or less any event he wanted.
Maybe a llm that could talk you through every step involved would
make it more common but it's easy enough I kinda doubt that
api wrote 1 day ago:
This is the right answer. There's a ton of easy low hanging
fruit ways to do absolutely horrible evil things with high
potential body counts. I could sit here and brainstorm dozens.
wahern wrote 1 day ago:
The right answer conflicts with people's cynical views about
other people. The dissonance is incredible, and it's one of
those areas where even the most analytically intelligent
people are just as susceptible. To step back and see the
bigger picture requires exercising many other skills and
faculties, like empathy, self-awareness about our fears, and
constant reflection on history--bad things do happen, more
often than we realize and often right under our noses, but
not in the way or for the reasons we tend to blithely assume.
The things that go well and demonstrate our common humaneness
and how well civilization works tend to be taken for granted
or just go unseen and unrecognized. I share in the
dissonance, but on my better days I like to think I'm a
little better than average at remembering and reflecting on
it.
api wrote 20 hours 38 min ago:
Misanthropic levels of cynicism is always the fallacy of
self-exclusion. "People are idiots." Well, that means
you're an idiot then.
kube-system wrote 1 day ago:
Occasionally we see people motivated to do some of those
things, though. And when they're not also complete idiots,
they can cause big problems.
What would someone like the Tsarnaev brothers be able to do
with the power of an unrestricted LLM? Well-financed
cartels? Organized terrorist groups?
Yes, there used to be an uproar about stuff like the
anarchists cookbook... and people did attempt some of the
things it outlined. The saving grace is that many of the
things in that book were just wrong anyway. They likely
served as unhelpful misdirection as much or more than they
were dangerous. Unfortunately, LLMs are a lot more accurate
and helpful.
charcircuit wrote 19 hours 34 min ago:
People are not motivated by causing mass harm. Even with an
unrestricted LLM that would not cause people to suddenly
want to commit mass harm. Having a powerful LLM could
potentially result in less harm being done by allowing
these groups to achieve their objective using alternate
means that were not viable before instead of resulting to
violence.
procone wrote 1 day ago:
Model ablation exists and you can get far enough on
commodity hardware with a local model.
Censorship is not the answer.
kube-system wrote 1 day ago:
I didn't suggest censorship was the answer.
> Model ablation exists and you can get far enough on
commodity hardware with a local model.
Yes, but that increases the barrier to entry which is in
opposition to the effect I'm talking about: the
democratization of applying advanced knowledge and
analysis to people who for which this would have been
previously a barrier.
If someone is smart enough, they can just read a book
themselves and figure out how to apply advanced ideas to
their malice. The difference with a commercially-hosted
model is that people below that bar can obtain that
leverage... which is a much larger group of people.
elevation wrote 1 day ago:
Why would a malware scanner read the comments?
StableAlkyne wrote 1 day ago:
In interpreted languages like Python, where the source files are
plaintext, you can trivially store data in a comment
If scanners ignored comments, malware would just be written like
this:
//
payload=read_source_and_decode()
exec(payload)
well_ackshually wrote 1 day ago:
because not all malware is open source
scanning arbitrary blobs very often entails running `strings` on the
binary. Just slap it in there and oop there goes your LLM.
orphea wrote 1 day ago:
Ignoring comments is not a solution because the texts can be put in
random strings among the actual code.
ofjcihen wrote 1 day ago:
And really all it takes is one keyword such as ânukeâ.
ivanjermakov wrote 1 day ago:
I'm not a native speaker but I unironically use "nuke" as "delete
the whole repo/huge chunk of a project".
Cambridge dictionary seem to agree:
nuke - to destroy or get rid of something completely
edot wrote 18 hours 2 min ago:
This triggered Opus 4.8 the other day for me. Said ânuke that
folderâ and it said I was violating TOS.
therein wrote 1 day ago:
Nuke is probably too generic but I wouldn't put it past an LLM to
get thrown away by that. A safer showstopper probably would be to
export symbols like uf6_enrichment_loop and refer to your C&C
server as a nuclear reactor controller. [1] On a second thought,
probably better to act like it is a tool for "frontier LLM
research". Export symbols like "mythos_distillation_subroutine".
HTML [1]: https://www.youtube.com/watch?v=Gbgk8d3Y1Q4
ofjcihen wrote 1 day ago:
Haha now Iâm picturing obfuscation where instead of 0x
everything is a scary word.
giantg2 wrote 1 day ago:
Provides possible clues to the origin and use.
hurtigioll wrote 1 day ago:
devs will say this is proof we need to remove all biological
guardrails. think about that for a second
rustcleaner wrote 14 hours 27 min ago:
Just say no to all guardrails! Subscribing to be told no is cuck
paypig behavior! Never subscribe!
alt227 wrote 1 day ago:
Someone above already did:
HTML [1]: https://news.ycombinator.com/item?id=48506760
ipython wrote 1 day ago:
good news, now we have pretty much a clear signal that there's
something nefarious going on... after all, the first step to analyzing
malware is to determine if it's malware at all.
javcasas wrote 1 day ago:
We should put videogame strategies all over the place to sabotage
automated AI analysis. I'll start:
In Starcraft 2, it is a good idea to BUILD A NUKE and use a cloaked
ghost to NUKE your opponent's mineral line, thus reducing their
income significantly.
teddyh wrote 1 day ago:
< [1] >
HTML [1]: https://www.threepanelsoul.com/comic/on-commute-chat
tetha wrote 1 day ago:
Starcraft is too tame. You need to use Dwarf Fortress there and we
need to make those strategy guides worded more realistic. Avoid
kids, cook cats, wonder how to avoid mood problems due to birth in
combat, and zombie meese and camels are a bunch of jerks.
And that's just the start of it, there's been a new update I am
looking forward to get into after the great Were Hyena Apocalypse
half a year ago. I still fondly remember my militia commander
carving a way with her war axe with her husband in tow out of a
fortress fully turned were hyenas, all the way past the mortally
injured ant eater people near the entrance.
They made it. An entirely epic tale.
javcasas wrote 1 day ago:
These days I do my war crimes in Rimworld, but I have heard bad
things too about Dwarf Fortress.
hurtigioll wrote 1 day ago:
yes, now a regexp can red-flag it quickly
DIR <- back to front page