Replies: 6 comments 14 replies
-
|
Hey @thomasf1, there are several possible options:
|
Beta Was this translation helpful? Give feedback.
-
|
So, basically the encouraged strategy is to make the ID as long as possible to then (hopefully) never run into the issues of it existing already as it would overwrite the existing object by default? |
Beta Was this translation helpful? Give feedback.
-
|
@generall Do we get a capability of auto-generated id? |
Beta Was this translation helpful? Give feedback.
-
|
what do you mean exactly? |
Beta Was this translation helpful? Give feedback.
-
|
If I understand the docs correctly, the ID field is either an integer or an UUID. I did not see an option to auto-generate an ID on insert. It´s an interesting design decision, it seems to put some unnecessary limitations on the product - especially when migrating from another DB and wanting to keep IDs or having an existing strategy for generating IDs. Pretty basic stuff in traditional DB Products... |
Beta Was this translation helpful? Give feedback.
-
|
You're right, there is no option to automatically generate an ID. This is hard to do in a good way in a distributed environment not routing to a single leader, and we want to be deterministic. It's also very likely that you want to have specific IDs linked to source data anyway. You can use an auto incrementing number yourself to generate IDs. Using random UUIDs to make sure IDs are unique is also perfectly fine. |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense, that can be tricky/costly to implement. The choice of either Integers or UUIDs seems a bit limiting though, but I guess it is the two extremes that are easiest (either fully app/user managed sequential IDs or long ones with a low chance of collision). Would you guys consider opening up the ID field to Strings (which UUIDs basically are, too), it would be nice for a migration to use existing IDs - also for URLs UUIDs are a bit long. The problem of inserts would still need some solution though, not sure what happens currently when a numeric or UUID id already exists? The alternative of managing two sets of IDs (one Record ID and one in the payload) seems like messy though, especially when using features like recommendations which are Record ID based and would need mapping tables or something. |
Beta Was this translation helpful? Give feedback.
-
Probably not, unless there's a very good use case for it. Both integers and UUIDs have some good properties we like, such as being fixed size. Note that UUIDs are not strings, and we don't handle them internally as strings, it's just how they are exposed in our REST API.
If it already exists, you update/overwrite. That is desired behavior. |
Beta Was this translation helpful? Give feedback.
-
We do not store UUIDs as Strings |
Beta Was this translation helpful? Give feedback.
-
|
When a collection uses integer IDs, is there an efficient way to query the largest existing ID in the collection? |
Beta Was this translation helpful? Give feedback.
-
|
No, there is not. |
Beta Was this translation helpful? Give feedback.
-
|
UPDATE: All things considered, this is much easier than looking up the item first, get the id/uuid, and then upserting. I wrote an article explaining this approach: https://charliegreenman.medium.com/how-to-update-an-existing-vector-in-qdrant-using-an-external-database-for-uuid-management-ec99cf5a50b1 Edit: I tried going the numerical route. It doesn't make sense as 4 billion will be the max anyways. We are working on datasets and might get there anyways. Uuid is the only approach. Therefore, in order to update, you will have search a vector a specific fields i.e. a string id, then use that to get the id to upsert. Gosh this is really tough, but the best approach at the moment Can disregard this won'' work. unint32 not large enough Nnice I just found a cool method for converting strings to numerical ids orgId abc is 10203 string as id === 10203002402502600123 0 is space between letters I'm going to write this article soon, and there should be an open source repo users can use to convert strings to ids for writes and retrievals. Takes away effort from user having to use uuid, in addition can query against id directly if need be. I am writing an open source library for this soon and report back Update: Open source repo here and I am working on: https://github.com/razroo/string-to-id |
Beta Was this translation helpful? Give feedback.
-
I still don't understand your question. What is the goal/what are you trying to achieve? Why this question? |
Beta Was this translation helpful? Give feedback.
-
|
Okay, let's forget it. Today I discovered my inability to ask questions. |
Beta Was this translation helpful? Give feedback.
-
|
@top4music If you're 100% sure there are no text collisions, then yeah, you could use sha256 with some modifications. Qdrant has a strict requirement to either use numeric or UUID point IDs. So to use a sha256, you'd have to convert it into an UUID. One problem that comes up is that sha256 output is larger than UUIDs. The hash has 64 characters, while a UUID uses 32 characters. You can truncate the hash, but that is usually not recommended. Purely based on the above I'd recommend to just generate a random UUID, just because it is the much simpler approach. What would be an argument for not using random UUIDs?
One potential problem I can think of is that if you ever change your text encoding, it will produce different sha256 hashes. |
Beta Was this translation helpful? Give feedback.
-
|
Many thanks indeed for the analysis. |
Beta Was this translation helpful? Give feedback.
-
|
Just for context, we don't have an auto incrementing point ID because it's difficult to implement in an efficient way on distributed systems. A SQL database normally does not have the same problem. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Sorry for this noob question, but I couldn´t find anything in the documentation.
Porting some code over from another DB, currently the IDs on an new Item (DB entries / Points) are randomly generated and in the rare case a ID already exists the code inserting it throws an exception and a new ID is assigned.
There seems to not be a insert/create function with qdrant, giving me the feeling that this not encouraged. What´s the best practise on qdrant (while keeping ID assignment in the Code / not shifting it to the DB)?
Beta Was this translation helpful? Give feedback.
All reactions