New Technologies cluster · Session 1 of 4
How language models are made, where they fail, and what they do to us
Łukasz Szoszkiewicz
Adam Mickiewicz University, Poznań
Global Campus of Human Rights
Monastery of San Nicolò, Lido di Venezia
The cluster
1
How models are made, where they fail, what they do to us
Day 1 · today
2
Find, cite and check in the UN Human Rights Database, then your first build in Garage Research
Day 1
3
The drafting and the votes: the Travaux and HRC Voting, then two builds
Day 2
4
UN recommendations (UHRI+), then a tool of your choice
Day 2
2 / 33
I
From bits to the next token
Part I · What the machine is made of
Every letter is a number, every number is 8 bits. Letters outside English (in red) take two bytes.
Drag the slider to zoom in, then hover over a pixel.
frame 0000 · Death in Venice (1971). 24 images a second; hover to stop on one.
A model turns each of them into a vector.
4 / 33
Exercise · 20 minutes
Round 1 · a human rights corpus, 5 sentences
Round 2 · the internet arrives, 5 more
Round 3 · training is over, one word moves
Watch: who is torn between two groups? Who did not move at all in round 1?
5 / 33
Round 1 · a human rights corpus
6 / 33
Round 2 · the internet arrives
7 / 33
Round 3 · training is over
Everyone else stays put: the map is now fixed. I read a new sentence; “court” steps towards the words around it, then goes back to its place.
The players walked onto the court.
The applicant addressed the court.
The same word, from the same place, gets two meanings depending on the sentence. How the model does it: attention →
8 / 33
The same corpus in a real model
↑ closer to the human rights sentences
closer to the everyday sentences →
torture
applicant
violation
treaty
asylum
court
convention
right
party
reservation
comics
tennis
restaurant
cake
left
● human rights · ● both · ● everyday
○ – – ● after round 1, then after round 2
Ambiguous words land in between, and move when the data change.
With little data the map is crude: after round 1, right and party sat exactly on asylum.
In the room your place was 2 numbers; in this model a word is 384. See one word →
paraphrase-multilingual-MiniLM-L12-v2. A word is the average of its sentences; its place is how close it is to each group of five.
9 / 33
What vectors are good for
Our 10 sentences: under a second, on a laptop.
All ECtHR judgments in English, 1.3 million paragraphs: under 2 hours, about $18. Try it in HUDOC Researcher →
One space for 50+ languages: a question in French finds the English sentence.
The result is a text that already exists (no text is generated).
| Query, in another language | Closest of our 10 English sentences | Similarity |
|---|---|---|
| La Cour a constaté une violation de la Convention. | The Court found a violation of the Convention. | 0.99 |
| Każde państwo-strona musi szanować prawo do azylu. | Every State party must respect the right to seek asylum. | 0.87 |
| Reservé una mesa en el restaurante para las ocho. | I made a reservation at the restaurant for eight. | 0.90 |
| Wir haben für Samstag einen Tennisplatz gebucht. | We booked a tennis court for Saturday morning. | 0.93 |
paraphrase-multilingual-MiniLM-L12-v2 · 118 million parameters · laptop, 11 Nov 2026 · Voyage pricing
10 / 33
Part I · What the machine is made of
GPT-2 (2019) runs in your browser. Type a sentence, move the temperature, then try your own language: count the tokens. Languages with less text online are cut into more pieces, and models tend to be weaker in them.
The model does not check facts but picks what usually follows such a text, which explains the confident tone and the invented sources.
11 / 33
Part I · What the machine is made of
Billions of sentences. The task: guess the next token. GPT-2 stopped here.
You: Can the police read my private messages?
GPT-2: On Wednesday, I met a group of journalists,
GPT-2 continues the text instead of answering.
People write thousands of pairs: a task and a good answer. On anything: conversations, legal letters, a language.
Task: Can the police read my messages?
Model answer: Only on a legal basis, usually with a court’s consent.
It learns the form. How we did it →
Two answers; a person picks the better one; the model is rewarded for it.
A: Yes, always.
B: It depends: a legal basis and oversight are needed. Chosen
It learns the style (careful and polite).
A conversation is only a format. The model still writes the next token, now as the assistant’s turn.
12 / 33
II
Brilliant at one task, surprisingly weak at the next
Part II · Uneven ability
1
2
1
2
■ what AI does well – – – tasks a person finds equally hard
INSIDE
OUTSIDE
1 Spotting an invented citation, with a database
93.5–100% caught
1 Spotting a real judgment that says something else
46–83% caught
2 Summarising a document it was given
77.2% of required elements
2 Recalling case law from memory, unchecked
some case numbers do not exist
Find your own frontier: ten questions from your work, with answers you know. Run every new model on them. How models are tested →
Dell’Acqua et al. 2026 · Liu, Stammbach and Henderson 2026 · VLAIR 2025 · Polish Supreme Administrative Court, I OSK 590/26
14 / 33
Part II · Uneven ability
2,047
court decisions in 41 countries on AI-invented content · Charlotin, to Sept. 2026
17–33%
wrong answers from Lexis+ AI and Westlaw · Magesh et al. 2025
My test · from memory
Article 54 of the ICCPR does not exist. 24 models said so in:
38%
of answers to a leading question
83%
to a neutral one
My test · search first, then answer (RAG)
How RAG works → Three ECtHR questions, checked in HUDOC.
| Cited case | no sources | with sources |
|---|---|---|
| correct | 31 | 36 |
| real, wrong detail | 7 | 1 |
| invented | 1 | 0 |
15 / 33
Part II · Energy and water
One text answer, in the data centre
Llama 3.1 8B (small, open)
measured on the chip ×2 · MIT
0.03 Wh
Gemini (median prompt)
measured · Google
0.24 Wh
ChatGPT (average query)
stated, no method · OpenAI
0.34 Wh
Coding agent: 5 messages, 24 model calls
estimate · Couch
~40 Wh
Log scale: each step is ten times more. A Gemini prompt is a 10 W LED for 1.4 minutes, and 0.26 mL of water.
Cheaper every year: Google’s energy per prompt fell 33 times in a year, and a small 2025 model beats 2024’s largest open model on a hard science test. But use grows faster: data centres took 1.5% of world electricity in 2025, about 3% by 2030; 40% of AI capacity sits where water is scarce.
The impacts fall on people living next to data centres; the UN Scientific Panel on AI (2026) finds that they “are growing significantly and are heterogeneous”.
Measure your own: CodeCarbon estimates the CO₂ of code on your computer.
= 0.24 Wh in the data centre
1.2 min
scrolling X
29 s
scrolling TikTok
11 s
streaming a film on TV
1.4 min
a 10 W LED bulb
Gemini: Google’s measurement. ChatGPT: OpenAI’s figure, no method. Llama: MIT, the chip measured and doubled. Agents: Couch’s estimate; his day is a few hours with 2–3 agents at once. An answer counts the data centre only; scrolling and streaming include the phone or TV and the network.
MIT Technology Review 2025 · Google 2025 · Altman 2025 · Epoch AI 2025 · Couch 2026 · IEA 2026 · Greenspector 2021 · IEA 2020 · UN Panel 2026
16 / 33
III
When the work runs through a model, what do we keep?
Part III · 1 · The evidence: Pakistan’s courts
Prompts per judge over 40 weeks: trained (▲) and access only (+). Fig. 5B
JudgeGPT: GPT-4 with a search over 128,292 judgments, given to 1,559 judges. A group drawn at random also got six lectures on using it. The experiment →
+6.3% more cases decided in a typical district: 1,848 a year. Slightly fewer appeals; no drop in the quality of judgments.
After training: fewer open legal questions, more editing and summaries, where the output is easy to check.
Not reported: how often JudgeGPT invented things.
Does working with AI teach you? Patent lawyers →
Mehmood, Goessmann and Ash (2026), Fig. 5B · Gent, IEEE Spectrum (2026)
18 / 33
Part III · 2 · Avoid or extend
The model’s answer becomes your answer, and you stop checking. If you cannot explain it, you have surrendered it. Offloading, deferral, surrender →
Junior patent lawyers wrote better drafts with AI, but showed no gain on a test without it.
Senior patent lawyers who worked with AI for three months did clearly better on a test without it. The study →
This cluster was made this way: about 40 hours of work with an AI coding assistant on the slides and Garage Research (31 October – 12 November), 75 saved versions and 295 test builds. The choices and the checks stayed with me.
A learning loop
try first → ask why → explain it back → try again. Not a boot loop: re-prompting until it works.
What it is good at
Teaching, when built for it: at Harvard, a GPT-4 tutor given the teachers’ step-by-step solutions guided 194 physics students instead of answering. They learned more than in class, in less time (not plain ChatGPT; one course).
What stays with you
Connecting the non-obvious. AI ideas converge: of 4,000 research ideas a model produced, 200 were unique.
Autor et al. 2026 · Shaw and Nave 2026 · Kestin et al. 2025, via AWorld on Bloom’s 2 sigma · Si et al. 2024
19 / 33
Part III · 3 · What to do
1
Write down your own answer before you ask.
A fluent answer makes it hard to imagine another one.
2
Use models to build tools, not as the analysis.
The code gives the same answer every time, and you can check it.
3
Cross-check: another model, another session.
Agents checking agents beat one agent: 93.0% against 88.9%.
4
Ask for options and arguments, and keep the verdict.
“Give me the pros and cons.” The decision stays with you.
5
If you would not sign it, it does not go out.
The responsibility stays with you.
6
Be intentional.
An agent can quiz you, interview you and show you its evidence, but only if you set that up.
7
Be honest about your place on the spectrum.
Can you explain the answer yourself?
8
A green check means one thing.
What did it check? What did it not check?
9
Read to know what decisions it made.
What did it decide for you?
10
Craft your prompt.
How you write your prompt shows how much you understood the problem.
1–4 from my own practice (3: Huang et al., DeepFact, ACL 2026) · 5–10 after Addy Osmani (2026)
20 / 33
Before
Decide
Prepare: search, read, write
Evaluate
With AI
Decide
1 your answer first
2 build tools
4 open questions
6 be intentional
10 craft your prompt
Prepare
the model’s part
Evaluate
3 cross-check
5 would you sign it?
7 can you explain it?
8 what did the check check?
9 what did it decide for you?
Use models where they are strong, and keep their harms small.
A warning from Bell and Legg (2026): writing is also thinking. The decision often forms while you write. · Purpose and task: Jensen Huang →
After the break
The UN Human Rights Database, built with AI. Then your first build in Garage Research.
A charged laptop, and one question from your studies or work that a General Comment may answer.
22 / 33
23 / 33
Appendix · How RAG works
Retrieval-augmented generation. Before the model answers, a search finds passages in a trusted collection, and the model must answer from those passages only.
You ask in your own words.
“Does facial recognition against a peaceful protester violate Article 8?”
A search engine picks the best passages from a fixed collection.
Here: 886 paragraphs of 23 ECtHR judgments.
Each passage keeps its source.
[1] Glukhin v. Russia, § 89
[2] Gaughran v. UK, § 68
[3] Libert v. France, § 42
The model gets an instruction, the passages and your question.
“Answer only from passages [1]–[3]. Cite each claim.”
Every claim points to a passage.
In Glukhin, facial recognition used to identify the applicant met no “pressing social need” [1].
Without RAG the model answers from memory, guessing the next token. With RAG it answers from sources you can open, and you still check the footnotes.
← Back to slide 15 · Is hand-built RAG only for now? →
24 / 33
Appendix · Patent lawyers
133 patent lawyers at 11 US firms, three months with or without an AI assistant, then a test without it.
Better drafts (+0.34 to +0.38 SD). Juniors gained the most.
Seniors who had used AI beat those who had not (+0.45 SD). Juniors: no gain on average.
Practise on your own first, as this is how you learn what to check.
Autor et al., “Does AI Assistance Enhance or Erode Expertise?”, NBER Working Paper 35720 (2026)
25 / 33
Appendix · Stage 2, from the inside
PLLuM is a Polish large language model built with public money by a consortium that includes NASK. In 2025 a team of lawyers wrote a dataset to tune it for law and public administration.
Administrative decisions (building permits, benefits, fines, residence permits), summons to remedy formal defects, transfers to the competent authority
Appeals against decisions, freedom of information requests, payment demands, consumer complaints, contracts, powers of attorney
One example = an instruction + the document a good lawyer would write. Two of the kind:
Access to information
“Ask the city office how many assemblies it banned in 2024, and on what grounds.”
→ a request under the Act on Access to Public Information
An effective remedy
“Appeal against a refused residence permit; the office ignored the applicant’s family.”
→ an appeal citing the Code and Article 8 ECHR
26 / 33
Appendix · Purpose and task
Machines take over tasks. The purpose of the job stays, and the job is rebuilt around it.
His example: reading scans is a radiologist’s task; diagnosing disease is the purpose.
When a task goes to a model, ask what it was for, and keep that.
27 / 33
Appendix · Offloading, deferral, surrender
You give the model a task. The result stays yours: you check it and can explain it.
The model summarises a judgment; you read the key paragraphs and fix the summary.
The model’s answer becomes the starting point of your next step, untested.
You build your argument on that summary without opening the judgment.
The model’s answer becomes your answer. You no longer feel the need to check.
The summary goes into your submission. Asked what the Court held, you cannot say.
The test: if you cannot explain the answer, even with the model’s help, you have surrendered it.
← Back to slide 19 · Addy Osmani (2026) · “cognitive surrender”: Shaw and Nave, Wharton (2026)
28 / 33
Appendix · What a word looks like to the model
court = [0.018, 0.062, −0.028, −0.053, −0.060, 0.001, 0.096, 0.022, 0.014, 0.044, … ] 384 numbers in all
Alone, court sits closer to tennis (0.53) than to violation (0.46). Without a sentence, the model cannot know which court you mean.
Compare any two: and
similarity –
In the room: 2 numbers. In this model: 384. In large language models: thousands for every token.
← Back to slide 9 · paraphrase-multilingual-MiniLM-L12-v2 · each word embedded on its own
29 / 33
Appendix · Attention: what round 3 was about
The
players
walked
onto
the
court
The
applicant
addressed
the
court
(Line thickness illustrates the idea; it was not measured from a model.)
Each token looks at the other tokens in the sentence and weighs how much each one matters to it.
It then mixes their meaning into its own vector: a tennis court, or a court of law.
This happens in dozens of layers. It is the T in GPT, the Transformer. GPT looks only back, at earlier words; models like BERT look both ways.
Vaswani et al., “Attention Is All You Need” (2017)
See it inside GPT-2: Transformer Explainer
30 / 33
Appendix · How models are tested
Artificial Analysis runs many public tests on each new model and combines them into one Intelligence Index, with price and speed beside it.
Read a score like evidence: which tasks? which settings? who ran it? Could the questions have leaked into the training data?
PrinzBench: a US lawyer’s own legal research test, 33 questions never published.
Secret, so the questions cannot leak into training data. However, nobody outside can check the score.
In June 2026, about half a year after its launch, the best model scored 91 of 99: the benchmark was saturated and no longer separates the strongest models.
Build your own: ten questions from your work, with answers you know. Run every new model on them.
31 / 33
Appendix · Is RAG only for now?
Rich Sutton (2019): over 70 years of AI research, general methods that grow with computing power beat human knowledge built in by hand.
My guess: hand-built RAG may not be needed in 3 to 5 years. Nevertheless, the question of sources will stay.
Appendix · JudgeGPT: the experiment
GPT-4 with one instruction for judges: “cite only the cases you were given”.
It picks its own search words; a keyword search runs over 128,292 judgments and 943 statutes and attaches the 10 best passages. Every answer has footnotes.
| Group | Judges | What they got |
|---|---|---|
| A | 1,197 | JudgeGPT and six lectures on using it |
| B | 180 | JudgeGPT and a general course |
| Control | 182 | The general course only |
The question: how much comes from access to the tool, and how much from learning to use it?
← Back to slide 18 · Mehmood, Goessmann and Ash, “Courts of Tomorrow”, CEPR DP 21783 (2026), pp. 10–13
33 / 33