Errors during setup of local filenode

Hi All,

I am in the process of setting up a self-hosted Anytype instance. I am using the Ansible scripts made available at GitHub - anyproto/ansible-anysync: ansible roles for any-sync-* daemons.

I generated the necessary configuration files using the anysync tool and modified the necessary parameters in the example files provided by the Ansible repo. I have reverted to the original keys for testing purposes to ensure that there are no typos on my end. All services and databases are running within the same server. The OS is Fedora Linux 39.

I am using a podman setup to create and satisfy the Redis, Mongo and S3 backend requirements. All of those components are up and running successfully.

The Ansible playbook runs successfully but I am unable to start the any-sync-filenode service. Here is what I see in the logs:

/bin/any-sync-filenode -c /etc/any-sync-filenode/config.yml
2024-02-13T02:09:52.157-0500 INFO common.nodeconf net configuration applied {“netId”: “N6jGqb6SfHa3aENsCjvnSMXFR9F3eZUhCCziqs28rkUyJRWw”, “id”: “65cb11d062a7428d898af579”}
2024-02-13T02:09:52.160-0500 INFO common.net.secure secure service init {“peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”}
2024-02-13T02:09:52.161-0500 DEBUG net.peerservice dial {“rootOp”: “periodicCall”, “peerId”: “12D3KooWRqFJtDMstqtcYAKZYEuGksD8A5WTHJZsFFk5beyrCVsq”, “addrs”: [“192.168.1.5:1020”, “quic://192.168.1.5:2020”]}
2024-02-13T02:09:53.292-0500 DEBUG common.net.peer peer close {“peerId”: “12D3KooWRqFJtDMstqtcYAKZYEuGksD8A5WTHJZsFFk5beyrCVsq”}
2024-02-13T02:09:53.292-0500 FATAL main can’t start app {“error”: “can’t run service ‘filenode.redisprovider’: read: connection reset by peer”}

I have noticed that every time I start the filenode, it triggers the error message below on the coordinator. The redis server itself is never contacted.

Feb 13 02:09:52 testsrv any-sync-coordinator[13905]: 2024-02-13T02:09:52.175-0500 DEBUG common.net.peer peer close {“peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”}
Feb 13 02:09:52 testsrv any-sync-coordinator[13905]: 2024-02-13T02:09:52.175-0500 DEBUG common.net.peer peer close {“peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”}
Feb 13 02:09:52 testsrv any-sync-coordinator[13905]: 2024-02-13T02:09:52.176-0500 DEBUG common.net.drpcserver drpc serve peer {“remoteAddr”: “192.168.1.5:39844”, “localAddr”: “192.168.1.5:1020”}
Feb 13 02:09:52 testsrv any-sync-coordinator[13905]: 2024-02-13T02:09:52.178-0500 INFO rpcLog {“totalDur”: 0.000021284, “addr”: “yamux://192.168.1.5:39844”, “app”: “v0.3.6”, “peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”, “identity”: “A6U8GvABVcLSBpuCV8AWbb2HF4MsJMZrveE1R8CzrazzHLsX”, “rpc”: “coordinator.networkConfiguration”, “peerVersion”: “any-sync-filenode:v0.5.4/any-sync:v0.3.7”}
Feb 13 02:09:53 testsrv any-sync-coordinator[13905]: 2024-02-13T02:09:53.293-0500 INFO common.net.peer serve connection error {“error”: “manager closed: EOF”, “errorVerbose”: “manager closed: EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231”}

I am running the application versions below:
/bin/any-sync-coordinator -v
any-sync-coordinator
v0.3.6
build on 2023-11-01T10:38:36Z from HEAD at #3127c7e(clean)
/bin/any-sync-filenode -v
any-sync-filenode
v0.5.4
build on 2023-11-28T18:26:13Z from HEAD at #fa946d5(clean)
/bin/any-sync-consensusnode -v
any-sync-consensusnode
v0.1.1
build on 2023-09-14T11:20:55Z from HEAD at #f586008(clean)
/bin/any-sync-node -v
any-sync-node
v0.3.17
build on 2024-01-17T19:32:17Z from HEAD at #1d747f7(clean

Any suggestions regarding what could resolve this issue will be greatly appreciated.

Hi!

  1. First, make sure your Redis instance supports the Bloom module. Here’s an example of how you can run it using Docker any-sync-dockercompose/docker-compose.yml at main · anyproto/any-sync-dockercompose · GitHub
  2. Note the port to connect to Redis instance ansible-anysync/roles/any_sync_filenode/defaults/main.yml at main · anyproto/ansible-anysync · GitHub.
  3. If that doesn’t help, post the contents of your configuration /etc/any-sync-filenode/config.yml, as well as details of the Redis running
1 Like

Thanks for your feedback. I am running redis with the bloom module enabled using port 6379. I am using this docker image: docker.io/redis/redis-stack-server 7.2.0-v8.

Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.504 * Running mode=standalone, port=6379.
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.506 * RedisBloom version 2.6.11 (Git=unknown)
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.506 * Module ‘bf’ loaded from /opt/redis-stack/lib/redisbloom.so
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.508 * Server initialized
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.508 . The AOF directory appendonlydir doesn’t exist
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.509 * Creating AOF base file appendonly.aof.1.base.rdb on server start
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.509 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.510 * Ready to accept connections tcp
Feb 13 10:58:46 testsrv redis[115015]: 1:M 13 Feb 2024 15:58:46.510 . 0 clients connected (0 replicas), 921112 bytes in use

Here’s my filenode config file:


account:
peerId: 12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ
peerKey: x0iUgSGBqiyKBvlQl3T+LDaAwSA6J1Z/ebrCtuFawegkryS+P9tv/G7D/g3rwoFSaL4yz8G8eyElG/KhmiXLvw==
signingKey: x0iUgSGBqiyKBvlQl3T+LDaAwSA6J1Z/ebrCtuFawegkryS+P9tv/G7D/g3rwoFSaL4yz8G8eyElG/KhmiXLvw==
network:
networkId: N6jGqb6SfHa3aENsCjvnSMXFR9F3eZUhCCziqs28rkUyJRWw
nodes:

  • peerId: 12D3KooWRyBw2GnQskP8kUFD91Ja46ecfa33VXs8mcjVv3N4T1dQ
    addresses:
    • 192.168.1.5:1010
    • quic://192.168.1.5:2010
      types:
    • tree
  • peerId: 12D3KooWRqFJtDMstqtcYAKZYEuGksD8A5WTHJZsFFk5beyrCVsq
    addresses:
    • 192.168.1.5:1020
    • quic://192.168.1.5:2020
      types:
    • coordinator
  • peerId: 12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ
    addresses:
    • 192.168.1.5:1030
    • quic://192.168.1.5:2030
      types:
    • file
  • peerId: 12D3KooWG52DEyA8hk4UbdzEnfPfyBZ1MLjcN2Ts7cDQkoYHo1U1
    addresses:
    • 192.168.1.5:1040
    • quic://192.168.1.5:2040
      types:
    • consensus
      drpc:
      stream:
      maxMsgSizeMb: 256
      timeoutMilliseconds: 1000
      log:
      defaultLevel: ‘’
      namedLevels: {}
      production: false
      metric:
      addr: 0.0.0.0:8002
      networkStorePath: /var/lib/any-sync-filenode/networkStore
      quic:
      dialTimeoutSec: 10
      listenAddrs:
  • 192.168.1.5:2030
    writeTimeoutSec: 10
    redis:
    isCluster: false
    url: redis://192.168.1.5:6379?dial_timeout=3&read_timeout=6s
    s3Store:
    bucket: any-sync-files
    endpoint: http://192.168.1.5:9000
    forcePathStyle: true
    indexBucket: any-sync-files
    maxThreads: 16
    profile: default
    region: us-east-1
    yamux:
    dialTimeoutSec: 10
    listenAddrs:
  • 192.168.1.5:1030
    writeTimeoutSec: 10
1 Like

Thanks for the details! Let’s check that Redis port is available to filenode. Run the command nc -vz 192.168.1.5 6379 from the server running any-sync-filenode. You should see “succeeded” if the port is available.

I would also like to point out that deploying with Ansible is the best option for a distributed cluster across multiple servers. In your case, when everything is running on a single server, you can try the docker compose solution :slightly_smiling_face:: GitHub - anyproto/any-sync-dockercompose: docker-compose for testing any-sync

Thanks again, I am able to connect to the port.

nc -vz 192.168.1.5 6379
Ncat: Version 7.93 ( Ncat - Netcat for the 21st Century )
Ncat: Connected to 192.168.1.5:6379.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

I can certainly give Docker a go. I mainly went down the Ansible path because the docs mentioned that the Docker config was not yet production-ready.

Indeed, there is nothing wrong with Redis.
So let’s take a look at the contents of your inventory.ini in Ansible.

Sure, here’s inventory.ini file:

[all:children]
any_sync

[any_sync_node]
any-sync-node1 ansible_host=192.168.1.5
; any-sync-node2 ansible_host=10.10.0.2
; any-sync-node3 ansible_host=10.10.0.3

[any_sync_filenode]
any-sync-filenode1 ansible_host=192.168.1.5

[any_sync_coordinator]
any-sync-coordinator1 ansible_host=192.168.1.5

[any_sync_consensusnode]
any-sync-consensusnode1 ansible_host=192.168.1.5

[any_sync:children]
any_sync_node
any_sync_filenode
any_sync_coordinator
any_sync_consensusnode

[mongo]
mongo1 ansible_host=192.168.1.5
mongo2 ansible_host=192.168.1.5
mongo3 ansible_host=192.168.1.5

[redis]
redis ansible_host=192.168.1.5

Thanks! At the moment, deploying with Ansible is not compatible with single-server mode: each component of the any-sync network has to reside on a separate server. I will add support for single-server mode to the Ansible repository in the next few days. Stay tuned!

2 Likes

Very cool, thanks for working on this. I will be looking forward to testing things once that’s in place. Thanks again.

Thanks for waiting! Ansible deployment has been improved and added support for single-server mode. GitHub - anyproto/ansible-anysync: ansible roles for any-sync-* daemons

  1. In your case, use the following inventory.ini
[all:children]
any_sync

[any_sync_node]
any-sync-node1 ansible_host=192.168.1.5

[any_sync_filenode]
any-sync-filenode1 ansible_host=192.168.1.5

[any_sync_coordinator]
any-sync-coordinator1 ansible_host=192.168.1.5

[any_sync_consensusnode]
any-sync-consensusnode1 ansible_host=192.168.1.5

[any_sync:children]
any_sync_node
any_sync_filenode
any_sync_coordinator
any_sync_consensusnode

[mongo]
mongo1 ansible_host=192.168.1.5
mongo2 ansible_host=192.168.1.5
mongo3 ansible_host=192.168.1.5

[redis]
redis1 ansible_host=192.168.1.5

[s3]
minio ansible_host=192.168.1.5

Be sure to use MongoDB in Replica Set mode! This is a requirement of any-sync-consensusnode

  1. Then update group_vars/any_sync.yml by simply adding your values and deleting the any-sync-node2 and any-sync-node3 sections
2 Likes

Thanks for your help and how quickly you worked on this. I appreciate all the changes in the Ansible configs, it was really easy for me to set up the new environment based on all the port changes etc

Unfortunately, I’m getting the same error message as before with the filenode. I do not think the Redis server is getting contacted at all since I get the same message with the Redis server on/off. Does the filenode attempt to contact the S3 server before or after the redis server? Just in case, I have tried running with the Minio server on/off with the same results. Anything else you can think of?

I have mongodb running in replica set mode. To confirm, for the mongodb connection, is this the correct format for password enabled databases?

mongodb://any-sync-coordinator:mypassword@192.168.1.5:27001,192.168.1.5:27002,192.168.1.5:27003/?w=majority&authSource=coordinator

LOG: any-sync-filenode
/bin/any-sync-filenode -c /etc/any-sync-filenode/config.yml
2024-02-20T04:45:17.740-0500 INFO common.nodeconf net configuration applied {“netId”: “N6jGqb6SfHa3aENsCjvnSMXFR9F3eZUhCCziqs28rkUyJRWw”, “id”: “65d46f542fa68489c5ff6c4e”}
2024-02-20T04:45:17.744-0500 INFO common.net.secure secure service init {“peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”}
2024-02-20T04:45:17.745-0500 DEBUG net.peerservice dial {“rootOp”: “periodicCall”, “peerId”: “12D3KooWRqFJtDMstqtcYAKZYEuGksD8A5WTHJZsFFk5beyrCVsq”, “addrs”: [“192.168.1.5:1002”, “quic://192.168.1.5:2002”]}
2024-02-20T04:45:18.881-0500 DEBUG common.net.peer peer close {“peerId”: “12D3KooWRqFJtDMstqtcYAKZYEuGksD8A5WTHJZsFFk5beyrCVsq”}
2024-02-20T04:45:18.881-0500 FATAL main can’t start app {“error”: “can’t run service ‘filenode.redisprovider’: read: connection reset by peer”}

LOG: any-sync-coordinator
Feb 20 04:44:33 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:44:33.887-0500 INFO common.net.pool GC: removed 0; cache size: 2
Feb 20 04:44:37 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:44:37.253-0500 INFO common.net.peer serve connection error {“error”: “manager closed: EOF”, “errorVerbose”: “manager closed: EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231”}
Feb 20 04:44:51 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:44:51.753-0500 INFO rpcLog {“totalDur”: 0.00220708, “addr”: “yamux://192.168.1.5:48204”, “app”: “v0.3.10”, “peerId”: “12D3KooWRyBw2GnQskP8kUFD91Ja46ecfa33VXs8mcjVv3N4T1dQ”, “identity”: “AB4hKEe7etUfMPo9NghwGHjNuZvAr7TgpevVSVWYXbPUULaP”, “rpc”: “coordinator.deletionLog”, “peerVersion”: “any-sync-node:v0.3.20/any-sync:v0.3.20”}
Feb 20 04:45:03 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:03.883-0500 INFO common.net.pool GC: removed 0; cache size: 0
Feb 20 04:45:03 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:03.887-0500 DEBUG common.net.peer peer close {“peerId”: “12D3KooWG52DEyA8hk4UbdzEnfPfyBZ1MLjcN2Ts7cDQkoYHo1U1”}
Feb 20 04:45:03 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:03.887-0500 INFO common.net.pool GC: removed 1; cache size: 2
Feb 20 04:45:17 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:17.755-0500 DEBUG common.net.drpcserver drpc serve peer {“remoteAddr”: “192.168.1.5:39394”, “localAddr”: “192.168.1.5:1002”}
Feb 20 04:45:17 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:17.757-0500 INFO rpcLog {“totalDur”: 0.00001397, “addr”: “yamux://192.168.1.5:39394”, “app”: “v0.3.10”, “peerId”: “12D3KooWCHZk6EwaYtAFS81pGrn1t5CCgBtTVKSxzuJkLYix8zwQ”, “identity”: “A6U8GvABVcLSBpuCV8AWbb2HF4MsJMZrveE1R8CzrazzHLsX”, “rpc”: “coordinator.networkConfiguration”, “peerVersion”: “any-sync-filenode:v0.5.4/any-sync:v0.3.7”}
Feb 20 04:45:18 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:18.883-0500 INFO common.net.peer serve connection error {“error”: “manager closed: EOF”, “errorVerbose”: “manager closed: EOF\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:231”}
Feb 20 04:45:33 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:33.883-0500 INFO common.net.pool GC: removed 0; cache size: 0
Feb 20 04:45:33 testsrv any-sync-coordinator[1813918]: 2024-02-20T04:45:33.887-0500 INFO common.net.pool GC: removed 0; cache size: 2

LOG: Redis

1:M 20 Feb 2024 09:45:37.591 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:45:42.609 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:45:47.627 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:45:52.646 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:45:57.663 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:46:02.682 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:46:07.713 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:46:12.751 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:46:17.774 . 0 clients connected (0 replicas), 921112 bytes in use
1:M 20 Feb 2024 09:46:22.802 . 0 clients connected (0 replicas), 921112 bytes in use

nc -vz 192.168.1.5 6379
Ncat: Connected to 192.168.1.5:6379.
Ncat: 0 bytes sent, 0 bytes received in 0.04 seconds.

nc -vz 192.168.1.5 9000
Ncat: Connected to 192.168.1.5:9000.
Ncat: 0 bytes sent, 0 bytes received in 0.03 seconds.

1 Like

Hi! Glad for the feedback. From the looks of it, your Redis and Mongo Url look ok, but I haven’t tested the password authentication method for Mongo. I also made a small Mongo URL fix for any-sync-coordinator in the Ansible repository. Please do pull the new changes.
So, let’s try to bring up an environment of Redis using the following tested method:

  1. Create the file docker-compose.yml and fill it with the following content:
version: "3.9"

services:

  redis:
    image: redis/redis-stack-server:7.2.0-v6
    restart: always
    command: redis-server --dir /data/ --appendonly yes --maxmemory 256mb --maxmemory-policy noeviction --protected-mode no --loadmodule /opt/redis-stack/lib/redisbloom.so
    ports:
      - 6379:6379
    volumes:
      - ./storage/redis/:/data/
  1. Run docker compose up -d.
  2. Run or restart all other components such as Mongo, Minio and any-sync-* daemons
  3. Provide me logs from:
  • Redis container
  • any-sync-filenode daemon
  • MongoDB (one of three instances)