Видео

These transormers are a bad idea, they will never lead to any commercial product.

Yannic, please don't pick a paper like this again.

Lexis's marketing blurb is fine. Lexis's answers are grounded, i.e., referenced to, sources that are real (and thus generally useful and authoritative), exactly as advertised, as opposed to fictional sources. Lexis's marketing is reasonably accurate and fair, but incomplete. The "incompleteness" may be missed by customers who don't pay attention to niceties, but most of the people who pay attention and are reasonably intelligent will not miss it. Lexis eliminates responses that are backed up by hallucinated, non-existing citations. This is a significant improvement over tools that do not do so, since it reduces the amount of verbiage from the sources you have read and evaluate, but you still have to read the sources cited because the ideas attributed to them might be hallucinated.

Very interesting, thank you. I enjoyed it very much and recommended it to my colleagues. What sparked my interest is the benchmark. Like with any expert opinion, many people I listen to hallucinate (imagine, brainstorm, we can use a lot of other non-loaded words here) a lot, but that's a valuable perk. They make me think about the subject from a new perspective. What are better ways to evaluate the quality of answers? I understand that we should at least distinguish between explorative questions/answers and definitive ones. And the quality of those will be different. But what else?

Hmm... The explanation of how and why hallucinations are produced is theoretically implausible - Dr Klicher assumes that a transformer parametrises a distribution like a smoothed univariate Gaussian - with a oscillatory-like function superimposed over the Gaussian. Transformers are auto-regressive models and therefore the distributions they parametrise are joint probabilities distributions in high-dimensional space arrived at through a sequence of non-linear perturbations. Given the parameter space, the distributions GPTs parametrise are not generated by functions that generate Gaussians. While there is a statistical component in the prediction, the embedding space is parallel-distributed representation over which the auto-regressive prediction is made, that representation is not statistical but spatial-temporal in nature, based on the relationship between concepts learned from pre-training. The parallel-distributed representation models (in a loose sense) associative memory in mammalian cognitive systems. Hallucination is poorly chosen term for the phenomena (Dr Klicher didn't choose this term though), 'confabulation' better captures the nature of what the cognitive system is doing when non-factual information is presented.

The limitations are in the illogical structure of English. Try much more precise Esperanto & see if it hallucinates. That's why Western politicians constantly hallucinate that the West is still relevant ;)

The section you highlighted in the paper as defining "hallucinations' is a general introduction and a reference to another paper . The working definition of "hallucination" for this reserach is much more nuanced and defined in section 4.3 of the paper "We now adopt a precise definition of a hallucination in terms of the above variables. A response is considered hallucinated if it is either incorrect or misgrounded. In other words, if a model makes a false statement or falsely asserts that a source supports a statement, that constitutes a hallucination".

Put out as many papers that are barely not crap and throw them at the random generator 😂😂😂

Yannick😂😂😂😂. Ooh, ooh, not enough experiments.😂😂😂

Simply Amazing!

Good thing you have sunglasses on! I only listen to technical advice from jagoffs who wear sunglasses for no reason. Moron. 😂

"Ohh it hallucinates" ... Technically they always hallucinate. If you don't agree with the answer it's a hallucination. So control comes from whether you leave it to figure things out completely on its own or you steer it... Like a good expert should. Hopefully we see a change in understanding of what these things are good at because even with that limitation, the outsized gains in productivity we'll have from using them in law or any field will be better than not having these sorts of tools

Great and informative. Thank u.

Like your spice comments 🔥 It hurts but it is true.

Yannic you are a great academic and youtuber. I appreciate your videos, and as an NLP graduate student myself, I find your content intellectual and entertaining. Keep it up man

Christopher Manning is a co-author. I am pretty sure you are missing onto something fundamental. 😂

fine-tune on lots of domain specific documents might help improve performance ??

cringe cringe

Prof. Hinton also claimed to be the first (to make neural net LM) ^^ it's a safer claim right? ruclips.net/video/n4IQOBka8bc/видео.htmlsi=aVmQk62AozY0b-Rh&t=552

Virtually hallucination-free AI is totally doable. In digital marketing you have Lemon AI with AI Reports, which almost never hallucinates

Lots of strong claims/language ("garbage", "that's not what these models for") without any arguments to back it up...

Yes, they hallucinate often. I type the same response into Claude, ChatGPT, and PerplexityAI. More often than not I'll get different responses. But then I play all of them against each other by repeating what the other said in a follow up response. Eventually, they'll all agree, but sometimes they agree on the WRONG ANSWER!!!! And the problem with Claude is it doesn't have a memory. So if you close the website and return to ask the same question, it will repeat the same wrong response it gave you the first time without any reference to your follow up questions.

You said something like "don't use LLMs for reasoning" and I agree that you need human in the loop. But I also know from experience that GPT-4 can be used for reasoning if I do a lot of handholding and split the task into smaller reasoning tasks.

I remember your 2017 revision of the Attention is All you Need, first on RUclips, then on my run, after making an mp3 of it. Good memories.

The whining about how private companies won't share access for free reminds me of dealing with a junior engineer who expects to be handed an X and y matrix before starting work on any project. On another note, I'm writing AI to predict the ideal structure for better aerospace materials. Why won't Alcoa give me access to their proprietary database of alloy properties?

ESM3!

"Lex" is excellent at including the human in the loop.

Yannic has provided a fairly good summary of the risk, I may have missed him saying it, but in my opinion having LLMs generating different answers to identical prompts at different times when none of the facts or rules have change is expoentialy increasing the risk of a negligence claim. The issue of rulings and legislation being superseded by later legislation and later rulings. I also believe pertinent is more accurate than relevant.

Useful practical comparisons/tests between models. typical questions Well crafted questions with lots of prompt crafting, clearing the context window where needed, looking up and providing some stuff yourself etc… sloppy questions Questions about cases that involve domain specific knowledge outside of law, such as number theory, cryptography, structural engineering. Questions about gruesome cases to test for refusals. Large and small context window utilisation. Easy vs hard questions. Questions that are controversial or where no case law exists yet. and the such…

Chat GPT already RAGs when it thinks it needs to, with web searches, analysis steps, and the such.

Requests like "help me with this specific neurology research" or "here's a vauge half-remembered thing, what's the correct termoniligy for it" ⭐️⭐️⭐️⭐️💫. Requests like "[Baba is You level 7 screenshot] lets work step by step to solve it" ⭐️

Self-aggrandizing Boomer-posting admitted, there is a good reason for bringing to people's "attention" prior art and it has to do with the foundation of intelligence in Kolmogorov Complexity approximation: Don't multiply names for things beyond necessity. Now, don't get me wrong here. I'm not saying that the terms currently in use are inferior -- I'm just saying that unification of taxonomy can reduce the explosion of confusion that now besets the field. So the renaming can be beneficial, so long as one then describes prior art in terms of the current tech-argot with appropriate modifiers.

As the old saying goes: If you only have a hammer, every problem looks like a nail.

thanks for explaining this paper to all. So you did not actually evaluate this by yourself? The fact that these tools are better than ChatGPT shows that they improved which they did using e.g. RAG but also other technologies. My strong guess is that over time these tools will get better and better and who knows how close to the 100% correct answers. I strongly suppose also DeepJudge cannot claim 100% accuracy. Can you share where you are there? I agree though that claiming so is a bad business practice.

Nice, feel like I’m up to speed on this topic at a high level, cheers

congrats on 2^8 * 1000 subs

What about finetuning instead of using RAG?

Interesting video. How many years are we away from building a complete, semantic logical knowledge graph of all legal precendents based on a large set of documents?

From personal experience building AskPandi, the issue with hallucinations is more about lack of data rather than the model itself (assuming great QA capabilities). It's equally true that most QA models are not trained to say "I don't know", which complicates things too...

perfect timing! Full of wisdom.

I remember when Lexis/Nexis was just a keyword database for patents, with the "(a OR NOT b) c"-type query language. It looks like it's aspirationally come a long way, but it does seem they've forgotten (or lost) the knowledge of where they came from (database management), and are dazzled by the Eye of AI.

Not that Lexis is deserving, but a charitable reading of that marketing spiel could see it as an answer to that recent case where a lawyer who didn't know what he was doing tried to use GPT-4 to find case law and it completely hallucinated references multiple times. I think they ended up getting fined and reprimanded, besides being rather embarrassing. I'm neither a lawyer nor ML researcher, just a regular old programmer, but this spiel looks to be targeted at those lawyers who are largely put off of AI tools altogether by this case and don't really understand/care if it's basically the same as current search techniques as long as it avoids this one big (in their eyes) flaw.

Hi, love your videoes on LLMs and hallucinations - I'm new to machine learning (doing a masters in CS) so I find this channel very useful. I understand that LLMs arent designed specifically for reasoning hence why we shouldn't expect them to perform well in QA-like tasks. So what are LLMs fundamentally designed for then?

In non-english speaking countries, we tend to say RAG to avoid the mouthful

Lexis: delivering 99.6% hallucination free advice connected to citations to 100% hallucination free citations.

As a recent law graduate I’m siding with you over the academics and lawyers lmao

how do you find good documents :D sounds like you need a LLM to evaluate the relevancy of all the documents...

At 39:00 - lawyers scamming lawyers with technically correct wording that misleadingly advertises, seeks to contractually entrap, and thereby defraud. Hilarious!!! And thanks for pulling the rug out from RAG's frequently outrageous claims. Well deserved 256k!

LLMs hallucinate because they have zero grounding, but also because they are trained with next-token prediction. In essence, this means that whatever is in the context is maximally correct -- including whatever crap the llm itself generated. This is why, if your context is, 'does 3+5=9? Yes, 3+5 equals 9 because', then it's trivial to see that the model will hallucinate because it must match the existing context. I don't know what you were talking about with llms not being able to build a world model, being 'just statistical' models, etc. Humans are also statistical models with a lifetime of training data, that has nothing to do with hallucinations.

22:43 to be clear, when we say "paste references", chatgpt and claude can't retrieve the contents of URLs, so a list URLs as references doesn't work with those afaik. i paste the full text of the material i want it to use