Building Graph Based RAGs

Pros and Cons of Using Graphs for LLM RAGs, Node Embeddings

Check out the corresponding YouTube Video here:

Introduction

In recent months, the intersection of Graphs and Large Language Models (LLMs) has become a hot topic in the AI community.

The potential of combining these two powerful tools offers exciting possibilities, especially in the realm of Retrieval-Augmented Generation (RAG).

RAG systems enhance LLMs by integrating external knowledge sources to improve their responses.

This article explores the pros and cons of using Knowledge Graphs for LLM-based RAG systems and delves into insights gained from Jesus Barassa’s book, Building Knowledge Graphs.

Jesús Barrasa

The Book Structure

Building Knowledge Graphs offers a pretty comprehensive and detailed look into knowledge graphs. The book is divided into several important components:

1. Foundational Concepts and Ontology Design

• First, Jesus and the other author, kind of sell you on the idea, saying how different companies use it, and what they were able to accomplish with it. So you can get a feel for how these graphs should and could be used. Companies like Google, Airbnb and Lyft.

Democratizing Data at Airbnb

Introducing the Knowledge Graph: things, not strings

Amundsen — Lyft’s data discovery & metadata engine

2. Data Ingestion and Normalization

• One of the defining features for a knowledge graph is that they aggregate data from different sources and let people interface with it as if they are talking to just one database. Naturally, efficient data ingestion and normalization processes are crucial for integrating diverse data sources into a knowledge graphs. The book outlines techniques for transforming raw data into a structured format that fits the organizing principle.
• I will explain what an organizing principle if you scroll down.

3. Graph Databases and Query Languages

• The book provides practical insights into using graph databases and query language Cypher. These tools are essential for managing, querying, and analyzing the the knowledge graph.
• Thankfully, Cypher is very intuitive.
• I will share my notes below.

4. Real-World Applications

• The book showcases various real-world applications of KGs, from enhancing search engines to supporting decision-making in enterprises. These examples illustrate the versatility and impact of KGs in different domains.
• Other interesting use cases include, fighting crime, finding the identity of users from their online sessions, mapping dependencies of tasks or software libraries. And my favourite, semantic search and language generation.
• Definitely the most useful part of the book.

Thoughts

In my mind, this stuff holds incredible potential for creating lots of cool new features, especially combined with LLMs. The book doesn’t seem to be too aware of LLMs and doesn’t show any use cases for RAG etc. However, the man himself has been on YouTube talking about RAG Graphs. So I am planning to watch that soon. But regardless, this book was incredibly helpful in helping me catch up on everything GRAPHS and Knowledge Graphs!

Pros and Cons of Using Graphs for LLM RAGs

Pros

1. Knowledge Graphs can probably reduce hallucinations, similar to how vector databases can help ground LLMs with more concrete information.
2. Similar to vector databases and embeddings help with semantic search, graph databases offer similar semantic search but it’s more structural and topological.
3. Knowledge graphs can and do link together different data sources. Essentially giving your LLM access to more data. That’s more structured.
4. Your LLM can cite the actual source of the information. Which definitely helps with traceability and trustworthiness.

Cons

1. It requires significant effort in data curation, ontology design, and continuous updating to ensure accuracy and relevance.
2. Integrating KGs with LLMs can be technically challenging. It involves aligning the representation of knowledge in graphs with the probabilistic nature of LLMs, which can be a non-trivial task.
3. The integration of KGs can introduce additional computational overhead. This might impact the real-time performance of RAG systems, especially when dealing with large-scale graphs and complex queries.
4. If your data is garbage then obviously no amount of impressive engineering will yield amazing results. This is known as:

Poodoo in, poodoo out!

Or the more positive version.

Quality in, quality out!

Case Studies

AirBnb

The problem at Airbnb was the overwhelming growth of people and departments. And every department would make their own databases, tables, random data sources, dashboards etc.

This meant employees in other departments wouldn’t be able find that data, if they did, they didn’t know whether they should use it or not. Because they didn’t trust it was applicable. And they didn’t know who to ask.

The solution was the Dataportal, a unified search and discovery tool that enables easy searching across all data resources, provides rich context and metadata for each resource, and promotes transparency by showcasing employee and team contributions to the data ecosystem.

So essentially an internal website for AirBnb employees for all things data. And they kinda made it bit like a social media. Where employees had profiles and they could ask each other questions about datasets and so on. As you can imagine, huge success, productivity went up, data siloes were gone, big hooray.

Google

The problem at Google was searching for information online often required sifting through countless irrelevant results, deciphering ambiguous terms, and piecing together scattered facts. Traditional search engines rely heavily on keyword matching, failing to grasp the nuances of language and the interconnectedness of real-world concepts.

The solution, Google’s Knowledge Graph revolutionized search by understanding entities and their relationships.

It made it easier for users to go deeper or broader, showing things like “People also search for”. This helped people make unexpected discoveries more often. And show things that people asked frequently.

Lyft

Lyft built a similar thing to AirBnb, except open. The best way is to show you:

Amundsen

Lyft developed this framework Amundsen, a data discovery platform, to address the struggle to efficiently locate and utilize the right data for their work.

By leveraging metadata, which describes and provides information about data, Amundsen allows users to easily search, understand, and access relevant data within their organization.

This also helps organizations maintain compliance by identifying and tracking personal data throughout their data infrastructure.

Cypher or Graph Query Language

CRUD Operations in the Graph

Just like any data store, Cypher empowers you to create, read, update, and delete (CRUD) nodes and relationships in your graph.

• CREATE: Introduce new nodes and relationships to your graph.

CREATE (p:Person {name: 'Alice', age: 30})
CREATE (c:City {name: 'New York'})
CREATE (p)-[:LIVES_IN]->(c)

• MERGE: Intelligently add data, either creating new elements or updating existing ones if they match your criteria.

MERGE (p:Person {name: 'Bob'})
ON CREATE SET p.age = 35
ON MATCH SET p.occupation = 'Engineer'

• MATCH: Locate specific nodes and relationships based on patterns and properties.

MATCH (p:Person {name: 'Alice'})-[:LIVES_IN]->(c:City)
RETURN p, c

• DELETE & DETACH DELETE: Remove nodes and relationships, with DETACH DELETE ensuring a clean removal even if the relationship is connected to other nodes.

MATCH (p:Person {name: 'Alice'})
DELETE p

MATCH (p:Person {name: 'Bob'})-[r:LIVES_IN]->(:City)
DETACH DELETE r

• SET & REMOVE: Modify properties on nodes and relationships.

MATCH (p:Person {name: 'Alice'})
SET p.age = 31

MATCH (p:Person {name: 'Bob'})
REMOVE p.occupation

Intermediate Querying: Refining Your Search

Cypher provides a wealth of tools for filtering and shaping your query results:

• String Matching: Use STARTS WITH, CONTAINS, and ENDS WITH to find nodes with specific text patterns.

MATCH (p:Person)
WHERE p.name STARTS WITH 'Ka'
RETURN p

Logical Operators: Combine multiple conditions with AND, OR, and NOT.

MATCH (p:Person)
WHERE p.age > 30 AND p.city = 'New York'
RETURN p

List Membership: Check if a property value exists in a list using IN.

MATCH (p:Person)
WHERE p.name IN ['Alice', 'Bob', 'Carol']
RETURN p

Filtering Relationships: Exclude specific relationship types or properties with NOT.

MATCH (p:Person)
WHERE NOT (p)-[:KNOWS]->(:Person {name:'Bob'})
RETURN p

Graph-Local vs. Graph-Global

Some concepts to be aware of are:

• Graph-Local: Queries anchored to a specific node, exploring its immediate surroundings.
• Graph-Global: Queries that search the entire graph without a starting point.

Also make sure you are aware of neo4j functions, you can do a lot of computations in the cypher query.

Advice

My advice, don’t skip learning the cypher query, if you actually internalize it, you will be able to be much more effective in analyzing the core idea of each use case the book provides.

Not only that you will be able to come up with different use cases and modify the existing query into your use case.

Organizing Principles

Layer 1: Property Graph Model

The property graph model is a low level contract between the database and the users, where users aren’t just you and I, it’s all the software that’s using it and potentially the AI as well.

And I mean contract here, in the sense that, it’s a certain rigid structure that guarantees all parties will abide by.

And that structure is basically:

• There are nodes and relationships
• Nodes can have labels
• Nodes can have properties (key value pairs)
• Relationships can have labels
• Relationships can have properties
• Relationships can have directions

Layer 2: Ontology

The ontology, is a higher level contract between the data and the data consumer.

A good way to understand it, is to first learn what a taxonomy is. Taxonomy Taxonomy, in a broad sense is the science of classification, according to the encyclopedia.

And an example of this is the Biological Taxonomy that organizes all living things.

This kind of relationships can be verbalized as “is a type of”, e.g. Startfish is a type of echinoderm.

While taxonomies excel at classifying items, ontologies take this a step further. They not only categorize, but also describe the relationships and properties between those categories. This richer model allows for:

Ontologies that can express relationships beyond “is a type of.” They might include:

• “Part of” (a wheel is part of a car)
• “Has a” (a car has an engine)
• “Causes” (rain causes flooding)
• “Located in” (a city is located in a country)

Ontologies define characteristics of categories. For instance, an ontology about cars might specify that a car has properties like “model,” “year,” “color,” and “fuel type.”

The structured nature of ontologies allows computers to reason and infer new knowledge. If an ontology knows that all cars have engines, and a particular object is a car, it can infer that the object also has an engine.

Practical Applications of Ontologies

Ontologies are used in a wide range of fields:

• Artificial Intelligence: They enhance natural language processing, knowledge representation, and reasoning capabilities in AI systems.
• Semantic Web: Ontologies are essential for creating structured, machine-readable data on the web, making information more accessible and usable.
• Healthcare: They help standardize medical terminology and facilitate data exchange between healthcare providers.
• E-commerce: Ontologies improve product categorization, search, and recommendation systems.

Thoughts

I mean I think it’s pretty promising stuff. I think knowledge graphs can help us do some pretty cool stuff with LLMs.

…

Okay, I should have been doing this notes thing while reading. But now I’m just too bored to write it all out. If you really like this stuff. Go read the book, maybe the authors will actually teach it far better.

But I do wanna just jump into the more interesting stuff.

Semantic Search

Here’s my idea.

Essentially all of my chats with AI, are everything I am interested in and wanna know more about. So all of that chat between me and AI are all text right.

I do a little bit of processing, extract all of the concepts from each chat. I add some salience score to each concept.

Salience is how related that concept is to the entire document.

Okay then you have nodes called “Chats” and nodes called “Concepts”. And they are linked up. Now you get a bunch of taxonomical information and turn it into an ontology organizing principle.

And then a little bit of query magic. And you have yourself a semantic search that will always give you something.

Basically if direct search returns nothing, then I always wanna see something related.

Speaking of related.

Recommendations

I wanna see related chats, when I’m in a chat. Because maybe I am asking something I’ve asked before. To try all these ideas, I needed my own ChatGPT. So I made one.

It’s essentially a ChatGPT clone. So far it works and anybody can use it. And it’s free. I mean it’s free to use the app, but you have to put your own tokens into it. Call it BYOT, Bring Your Own Token.

Also it’s still in beta. So don’t expect like crazy stuff. Parts of it, I’m still working on.

Chat Tree

But, recommendations. What if when you’re in a chat, it offers other chats that are similar to it. Essentially that feature, “You may also like”.

That’s just a recommendation engine.

Basically similar idea to semantic search, key piece being here, the salience metric so that we can compute the similarity at query time and rank in descending order.

Now the other type of recommendation engine is collaborative filtering, but that one is more along the lines of “People like you also like”. But since our app is not really a social app, it doesn’t really make sense to do that.

GAG

You know since RAG or Retrieval Augmented Generation was so catchy, maybe GAG or Graph Augmented Generation will be too.

So what is that. Well remember how Retrieval Augmented Generation is searching the vector database for something, getting results. Then using those results to write up an LLM response.

Well same here. Just replace Vector Database with Graph Database.

However the problem is, the vector database really simplifies the query part. It returns the most relavant stuff. While the graph database offers a little bit of flexibility into the question “What’s the most relevant stuff?”.

I mean this concept already exists in GraphLand. It’s just plain old Question Answering.

We were previously looking for a Chat or Document, that matched our search query.

Now we are straight up looking for the Answer to our search query.

But to do that I you need to extract facts from the Chat. The book recommended diffbot, however I tried the platform and it was garbage so I decided to just use an LLM. Because it was a lot more debuggable.

But before I do that, let’s address some possible problems.

There are a few problems in LLMs right now.

Unreliable and Inconsistent Responses

Let’s say our task is “fact extraction” from text. Sometimes the LLM will miss a fact or two or three.

Well we can do sampling and voting.

But that actually solves a different problem, so we will do sampling and setting.

Meaning we will run the same task multiple times and collect the results. Then we will add each array item to a set. So that we only add unique items to the set.

DAG

Next problem is reliability. Imagine you have a DAG or Directed Acyclic Graph, or a chain or pipeline of AI tasks. And let’s say in the simplest scenario, of 3 different LLM tasks chained together.

At any step, the AI can fail, where failure is anything unexpected to even if its in the right format, just bad quality output.

So step 1 has 90% success.

Step 2 has 90% success.

Step 3 has 80% success rate.

While individual success rates are pretty high, the total success rate is only 65%.

So how can we fix this?

Feedback loops of course.

What if we add a validation step to each step. Where if the output is bad, then it recalls the same LLM with the same parameters, again and again.

Of course the problem here is possible infinite loop, but also slowing the whole process down, compounded by the fact that each step has to do this.

But then we can add a redesign stage. If your task success rate is anything below 80 to 90 percent, then just break up the problem so that it is more tasks.

Hopefully that will help solve most of our issues.

GAG

Back to GAGs, our evil plan worked perfectly so we now have facts extracted from the text. So we just gotta add it to our knowledge graph. And we have a fact graph.

We will use this shape for importing the facts.

Entity — Property — Value — Qualifier.

Then once that is all in there. We can query it.

Boom, we can extract facts from our knowledge graph.

But we’re not done yet.

GAG V2

Let’s also have these facts generate language for us.

You know how we use Subject Verb Object or SVO to talk in English. And you know how Yoda uses OSV.

“The greatest teacher, failure is.”

• Yoda

Here, qualifier is greatest.

Object is teacher.

Verb is is.

Subject is failure.

So from QOVS to SVQO.

Well we kinda already grabbed Entity Property Value, which is just another way of saying Subject Verb Object.

So we can say things like “Bob robbed the shop”. Which is in that shape. And you know how we also capture some optional qualifiers.

Here are a few examples of qualifiers in sentences:

Time:

• He is sometimes talkative.
• She always arrives early.

Frequency:

• I rarely eat fast food.
• They usually go for a walk in the morning.

Degree:

• She is a very talented musician.
• The coffee is quite strong.

Possibility:

• It might rain tomorrow.
• They are likely to win the game.

So that’s it. Our graph could technically generate language without even an LLM.

But we’re not gonna stop there.

We will get these basic factual sentences from our text. Then prompt the LLM to only rewrite better so that it sounds nice, without changing facts. Hopefully this makes it hallucinate a whole lot less.

Final Problem

Well you know how the text we fed into it in the first place was coming from our chats with an LLM?

So obviously no amount of clever engineering can get us quality output if the input is bad.

Remember the Poodoo Principle?

Pudu in Pudu Out

So we add WYSIWIG text editor into our chat platform.

WYSIWIG. Stands for What You See Is What You Get. This kind of editor is essentially the same UI as Notion.

The idea is you can now enter your own text, hopefully a piece of text that you trust and verified. And then it’s the source of your RAG, GAG, DAG, GAG2 and Chat.

More

Once you have facts extracted from a text in a nice structured format. You can do 3 obvious things:

• Turn it into flashcards
• Do sample and voting
• Do fact verification

Flashcards

I don’t have to tell you that learning and memorization is important and and we should be using flashcards.

But the process of creating them is cumbersome, while the effort of creating them obviously signals to the brain, this is important. We still may not do it because of the added barrier, after all we’re all lazy bums.

So get the LLM to use the list of facts to generate flashcards. Then we are given a spaced repetition session, multiple times, the system removes the stuff you know, stuff you don’t know it will just keep showing you I guess.

Sample and Voting

I think, I’m not sure, but we could do sampling and voting with a same prompt to try and generate the same fact.

Remember that sampling and voting was kind of a hazy process, where the document with the most similarity to the others “won”. Well now we can kinda open each document a list all of the “facts” or let’s call them “claims” for now.

Then we can do more semantic checking of the truthness of documents.

Which brings us to fact checking.

Fact Verification

I guess we do sampling and voting on the internet data. Because we basically can’t just assume anything is true.

We can only find consensus.

This makes me think of a new way of SEO, where something becomes true by way of frequency, and not logic.

Like some bad actor, will post some information that’s not true, just everywhere they can, while knowing that AI systems will find all of those, sample and vote, and now that becomes the truth.

But yeah, we grab documents off the internet and do the previous process of sampling and voting.

To try and find the true facts, however murky that may be.

In the future, it’s clear that we should be doing weighted sampling and voting. And weight the data sources with formal verification systems way more, and do a different kind of fact checking within those systems. Logic, math and science based.

That’s it

I was having some issues with Neo4j’s own hosted database service, I think it’s called Aura, because they don’t allow you to install libraries like neosemantics, which is a third party library, which I needed. So I had to install my own Neo4j, so I had to thinker with some Docker and do some flask backend.

Also I am gonna add text embeddings layer into it. Which should help with the question answering and semantic search.

And finish it up and hook it up to the fronted. So hopefully I can finish that in the next few days. Test it a bit and release.

So if you wanna update on all of this. Subscribe and follow on your favorite social media. And I will update, once it’s up.

Bye!

References

https://docs.google.com/document/d/149lPkXCo3KrcCRZU543pgfORuVPADTmrLUMvkgpjT3Y/edit?usp=sharing