Bad indexing of pages since the update

[thisisAlex]

Come on, guys! This is a serious issue.

I was about to try VACCUMing the SQLite files. But then, I decided I’d rather just rename my local anytype folder, and sync again (if you can’t rename it, even when you have closed Anytype, use Task Manager to fully kill it).

And still

I try to search IN THE CURRENT VIEW and

Maybe I’ll reinstall and retry. If you need volunteers to try things, I’m available.

Chat is cool. Like really cool. And I love it’s here. But a broken search may be a deal breaker for newcomers. Please, address this ASAP.

[dudedade]

had the same issue, today I found the new version 0.50.5 and now it works again!

[vincentl]

I have the same annoying problem. The search does not produce consistent results. It finds some objects but not others.
The new version 0.50.5 doesn’t change anything for me.

[requilence]

@thisisAlex @vincentl Yes, we do need volunteers to debug search problems, as we cannot reproduce them. You can contact me at [email protected] or @requilence on Telegram. Please attach “Device ID” from Help - Technical information menu.

[requilence]

Also, can you try one thing? If you open the object you cannot find, are you able to find it afterward? What if you make some changes to it, like adding an empty line?

[dudedade]

no sorry, that was too fast…does not work. some objects are missing and are not listed in the search

I tried what you suggested, to go into missing object and modify, but it doesn’t change anything. Same object remains unfound

[Ronsox]

I am facing this issue since several version on my Windows PC.
Mac and Iphone are fine.

This issue is really weird and destroys the whole experience.
I also don’t understand why and when objects are found.

I had the problem, that “Customers” I created could not be found.
After an update I though, cool they fixed it as I was able to search for them now.
After I edited some of the Objects I recognized, that these Objects cannot be found anymore.
Other Objects could still not be found after the update but when I create a new Object I can find it.
It does not matter if I search for them by ctrl+k oder within the grid view of the type.
Even if the search does not work, the objects are shown in the list of last objects.
Another thing is that I can search for an Object that has been deleted and also cleared the bin.

So in my experience we have following behaviors:

a) editing an object can cause make them findable but also make them not findable anymore (or nothing changes)
b) although an object has been removed, I still can find it with the search

Updating an Object will make it findable

Updating an item can make it not findable anymore (cannot reproduce it)

Sorry for having to redact so much, but this is sensitive data.

Find deleted Objects (bin cleared all properties removed)

[requilence]

I’ve created a debug utility to diagnose the search problem: https://github.com/anyproto/tantivy-dump

Do this (Windows):

1. Download tantivy-dump-x86_64-pc-windows-msvc.exe and keep it in Downloads. You can also build it yourself from the source code if you prefer.

2. In Anytype, copy your Account ID from Help → Technical information.

3. Open PowerShell and paste (edit the two lines at the top only). For the KEYWORD put the exact part of object’s title or body (make sure it has the same case). If you have object “Reading a Book“ you can put “Reading“

$ACCOUNT_ID = "YOUR_ACCOUNT_ID_HERE"
$KEYWORD    = "YourExactKeywordHere"

$DOWNLOADS = Join-Path $env:USERPROFILE "Downloads"
$EXE       = Join-Path $DOWNLOADS "tantivy-dump-x86_64-pc-windows-msvc.exe"
$INDEXDIR  = Join-Path $env:APPDATA "anytype\data\$ACCOUNT_ID\fts_tantivy\16"

Write-Host "`n===== KEYWORD (exact match, case-sensitive) =====`n"
& $EXE $INDEXDIR --keyword $KEYWORD

Write-Host "`n===== SEARCH (broader match) =====`n"
& $EXE $INDEXDIR --query $KEYWORD --fields Title --tokenizer-lang en

As an output you should see something like this:

It will be very helpful if you will share the output here or in DM.

[Ronsox]

Ok, so I did this with an Object that will be found within Anytype and with an Object that cannot be found.

These are my “Cars” Objects:

Output for Object (Ford) that can also be found within AnyType:

Output for Object (Mercedes) that can NOT be found within AnyType:

And yes, it can be found on iPhone & Mac

[vincentl]

OK, I have one object called ‘Teaming up for project’.

If I search for the keyword ‘Teaming’, this is the result: not in the list but it has found 4 other related objects.

[requilence]

@vincentl @Ronsox Thanks a lot! It means this exact titles were not indexed. Let’s check what you have in the index for these objects then.

1. Get the ID of the problematic object. The easiest way is … Menu on the top right corner → Advanced → Copy deep link. In the link, you will see objectId=baxxx& → copy baxxx.
2. Perform a search for this object ID.

$ACCOUNT_ID = "YOUR_ACCOUNT_ID_HERE"
$KEYWORD    = "OBJECT_ID"

$DOWNLOADS = Join-Path $env:USERPROFILE "Downloads"
$EXE       = Join-Path $DOWNLOADS "tantivy-dump-x86_64-pc-windows-msvc.exe"
$INDEXDIR  = Join-Path $env:APPDATA "anytype\data\$ACCOUNT_ID\fts_tantivy\16"

& $EXE $INDEXDIR --keyword $KEYWORD

You will see the results with different blocks and properties from this object. I want to understand if you have any info, is it outdated, what title do you have there(stored in doc id <objectid>/r/name)

[vincentl]

Interesting it seems that the objectID isn’t referenced at all

I did the same manipulation with an object that I found in the search.

[thisisAlex]

C:\Users\uXXXXXX\Downloads\anytype-debug>.\tantivy-dump-x86_64-pc-windows-msvc.exe %INDEXDIR% --keyword %KEYWORD%
No documents contained the substring "Pavel" in stored fields (path: C:\Users\uXXXXXX\AppData\Roaming\anytype\data\XXXXXXXXX\fts_tantivy\16).

C:\Users\uXXXXXX\Downloads\anytype-debug>.\tantivy-dump-x86_64-pc-windows-msvc.exe %INDEXDIR% --query %KEYWORD% --fields Title --tokenizer-lang en
No documents matched query "Pavel".

C:\Users\u1356989\Downloads\anytype-debug>

Can’t find Pavel

[Ronsox]

This is the result when using the Object_ID

Output for Object (Ford) that can also be found within AnyType:

Output for Object (Mercedes) that can NOT be found within AnyType:

Side info: Both deeplinks work and will route to the correct Object.

[云之巅]

The 0.50.5 version still has this issue. It’s really uncomfortable to not be able to find the content of this object despite its existence.

[vincentl]

Update 29/10

No finally I still can’t find all objects. The search results has improved somehow, however, I still can’t find all objects. Even the newly objects I’ve just created are missing.

---

It might help to know this. Since installing the latest Windows 11 patch, the Anytype search function has worked properly again.
Apparently, MS has fixed some localhost access issues. I don’t know if this is the reason why the Anytime app works as it did before.

[cicko]

Simple things, like an ability to reindex the space manually, would probably do wonders here.

[dudedade]

Hi Team

Is there any update on dev progress?

[thisisAlex]

@cicko proposal would be great. I bet that while the final solution is here, having at least the way to manually trigger a reindexing could save the day.

In the meanwhile, is there any workaround that we can try? I sometimes have to resort to search on my Android app and then go back to the desktop app.

(suddenly, an idea pops in my mind)

I might have come across a quick ‘n dirty reindexing process:

• Close Anytype. Make sure it’s fully closed (after quitting, check Task Manager for any remaining instances of Anytype.exe and close them, if needed)
• Go to the <ANYTYPE DATA DIR>\data\ folder
• Delete the fts-tantivy folder. (I always backup the full Anytype folder first, just to be safe)

As soon as I restarted, the folder got recreated, and search seems to be working fine.

By the way, the backup folder takes 4 GB, while the new one, just below 600 MB.

<insert here the due disclaimer about how I hold no liability for you, tampering with your Anytype folder >

[cicko]

My current PKM tool is SiYuan and they have tons of useful options and configurations, like the two above.

I guess their target audience is a bit different and the editing functionality seems the most complete, compared to other similar tools I’ve tried so far. Sadly, they don’t seem to be interested in going down the CRDT / p2p sync route.