Developers Forum for XinFin XDC Network

xu zhaolin
xu zhaolin

Posted on • Updated on

Graphnode Xinfin Compatibility Project (Done)

Dear all,

We have finished all the work that needed for graphnode to be compatible with XinFin netwwork. You can duplicate our steps to get the same results!
**
We have modified some of the graph source code to make it compatible with xdc network! you must use the code we provide to implement our method.**

https://github.com/gzliudan/graph-node
Graphnode overview,we have modifeid it to be compatitable with the xdc!!!!

https://github.com/gzliudan/parity-common
gzliudan/parity-common: Collection of crates used in Parity projects

https://github.com/gzliudan/bad-token-subgraph
The subgraph for XRC20 on blockchain XinFin and Apothem

https://github.com/gzliudan/stake-subgraph
The subgraph for StakeContract on blockchain XinFin and Apothem

Step-by-Step tutorial

Deploy the Graph

Deploy three daemons:IPFS, PostgreSql, and graph-indexer. Ipfs and graph-indexer should run with the current user right.

1、Install IPFS

full tutorial: https://docs.ipfs.io/install/command-line/#official-distributions

1.1 Modify UDP Receive Buffer Size

Tutorial: https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size

sudo sysctl -w net.core.rmem_max=2500000
echo "net.core.rmem_max=2500000" | sudo tee -a /etc/sysctl.conf
Enter fullscreen mode Exit fullscreen mode

1.2 Instal IPFS

Tutorial: https://docs.ipfs.io/install/command-line/#official-distributions

wget https://dist.ipfs.io/kubo/v0.14.0/kubo_v0.14.0_linux-amd64.tar.gz
tar -xvzf kubo_v0.14.0_linux-amd64.tar.gz
cd kubo
sudo bash install.sh
ipfs --version
Enter fullscreen mode Exit fullscreen mode

1.3 Initialize IPFS

export IPFS_PATH="/var/lib/ipfs"
echo 'export IPFS_PATH="/var/lib/ipfs"' >> ${HOME}/.bashrc

USER="`whoami`"
sudo mkdir -p /var/lib/ipfs
sudo chown -R ${USER}:${USER} ${IPFS_PATH}
sudo chmod ug+rwx ${IPFS_PATH}

ipfs init
cp -p ${IPFS_PATH}/config ${IPFS_PATH}/config.bak
sed -i "s#/ip4/127.0.0.1/tcp/5001#/ip4/0.0.0.0/tcp/5001#g" ${IPFS_PATH}/config
ipfs config --json API.HTTPHeaders.Access-Control-Allow-Origin '["http://192.168.2.162:5001", "http://127.0.0.1:5001"]'
ipfs config --json API.HTTPHeaders.Access-Control-Allow-Methods '["PUT", "POST"]'
Enter fullscreen mode Exit fullscreen mode

1.4 Creat Service

sudo tee /etc/systemd/system/ipfs.service <<EOF
[Unit]
Description=InterPlanetary File System (IPFS) daemon
Documentation=https://docs.ipfs.tech/
After=network.target
Requires=network.target

[Service]
Type=simple
User=${USER}
Group=${USER}
Environment="IPFS_PATH=${IPFS_PATH}"
Environment="IPFS_FD_MAX=8192"
LimitNOFILE=8192
MemorySwapMax=0
# TimeoutStartSec=infinity
WorkingDirectory=${IPFS_PATH}
ExecStart=/usr/local/bin/ipfs daemon
RestartSec=10
Restart=always
KillSignal=SIGINT
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable ipfs
sudo systemctl start ipfs
sudo systemctl status ipfs
curl -X POST http://127.0.0.1:5001/api/v0/version
# response: {"Version":"0.14.0","Commit":"","Repo":"12","System":"amd64/linux","Golang":"go1.18.3"}
Enter fullscreen mode Exit fullscreen mode

2、Install PostgreSQL

Tutorial: https://www.postgresql.org/download/linux/ubuntu/

  • database username: graph
  • password: daniel2022
  • listener ports: 5432
  • database name: apothem_db、xinfin_db

2.1 Modify the kernel parameter

cat /proc/sys/vm/swappiness
sudo bash -c "sysctl -w vm.swappiness=1"
echo "vm.swappiness=1" | sudo tee -a /etc/sysctl.conf
cat /proc/sys/vm/swappiness
Enter fullscreen mode Exit fullscreen mode

2.2 Install language support(i use Chinese here)

sudo apt install -y language-pack-zh*
Enter fullscreen mode Exit fullscreen mode

2.3 Install software

# Create the file repository configuration:
sudo sh -c 'echo "deb [arch=$(dpkg --print-architecture)] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

# Import the repository signing key:
# wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/postgresql.asc

# Update the package lists:
sudo apt update

# Install PostgreSQL 14
sudo apt install -y postgresql-14 postgresql-client-14 postgresql-server-dev-14 libpq-dev
Enter fullscreen mode Exit fullscreen mode

2.4 Backup config file

cd /etc/postgresql/14/main
sudo cp -p pg_hba.conf pg_hba.conf.`date +%Y%m%d-%H%M%S`
sudo cp -p postgresql.conf postgresql.conf.`date +%Y%m%d-%H%M%S`
Enter fullscreen mode Exit fullscreen mode

2.5 Modify pg_hba.conf

sudo tee /etc/postgresql/14/main/pg_hba.conf <<-\EOF
# Database administrative login by Unix domain socket
local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     md5
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            md5
host    replication     all             ::1/128                 md5
EOF
Enter fullscreen mode Exit fullscreen mode

2.6 Some parameters adjustments

sudo tee /etc/postgresql/14/main/conf.d/liudan.conf <<-\EOF
listen_addresses = '*'
port = 5432

tcp_keepalives_idle = 300
tcp_keepalives_interval = 20
tcp_keepalives_count = 9

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_line_prefix = '%m:%r:%u@%d:[%p]: '
log_rotation_age = 1d
lc_messages = 'en_US.UTF-8'            # log is shown in English

shared_preload_libraries = 'pg_stat_statements'
EOF
Enter fullscreen mode Exit fullscreen mode

2.7 change os user's postgres password

echo postgres:daniel2022 | sudo chpasswd
Enter fullscreen mode Exit fullscreen mode

2.8 start the database

# pg_ctlcluster 14 main start
sudo systemctl daemon-reload
sudo systemctl restart postgresql@14-main
systemctl status postgresql@14-main
Enter fullscreen mode Exit fullscreen mode

2.9 change database user's postgres password

su - postgres
psql -c "ALTER ROLE postgres WITH PASSWORD 'daniel2022';"
exit
Enter fullscreen mode Exit fullscreen mode

2.10 Creat Database

  • create user of the database: graph
  • create database: apothem_db and xinfin_db
PGPASSWORD=daniel2022 psql -X -h 127.0.0.1 -U postgres postgres <<-\EOF
CREATE USER graph WITH PASSWORD 'daniel2022' nocreatedb;
CREATE DATABASE apothem_db WITH OWNER=graph TEMPLATE=template0 LC_COLLATE='zh_CN.UTF8' LC_CTYPE='zh_CN.UTF8' ENCODING='utf8';
CREATE DATABASE xinfin_db WITH OWNER=graph TEMPLATE=template0 LC_COLLATE='zh_CN.UTF8' LC_CTYPE='zh_CN.UTF8' ENCODING='utf8';
EOF
Enter fullscreen mode Exit fullscreen mode

2.11 Create extension

create extension: pg_trgm、pg_stat_statements、btree_gist、postgres_fdw

2.11.1 testnet apothem

PGPASSWORD=daniel2022 psql -X -h 127.0.0.1 -U postgres apothem_db <<-\EOF
CREATE EXTENSION pg_trgm;
CREATE EXTENSION pg_stat_statements;
GRANT ALL ON FUNCTION pg_stat_statements_reset TO graph;
CREATE EXTENSION btree_gist;
CREATE EXTENSION postgres_fdw;
GRANT USAGE ON FOREIGN DATA WRAPPER postgres_fdw TO graph;
EOF
Enter fullscreen mode Exit fullscreen mode

2.11.2 production net xinfin

PGPASSWORD=daniel2022 psql -X -h 127.0.0.1 -U postgres xinfin_db <<-\EOF
CREATE EXTENSION pg_trgm;
CREATE EXTENSION pg_stat_statements;
GRANT ALL ON FUNCTION pg_stat_statements_reset TO graph;
CREATE EXTENSION btree_gist;
CREATE EXTENSION postgres_fdw;
GRANT USAGE ON FOREIGN DATA WRAPPER postgres_fdw TO graph;
EOF
Enter fullscreen mode Exit fullscreen mode

2.12 setting up environment for psql :

cat >> ${HOME}/.bashrc <<-\EOF
export PGHOST=127.0.0.1
export PGPORT=5432
export PGDATABASE=apothem_db
export PGUSER=graph
export PGPASSWORD=daniel2022
EOF

source ${HOME}/.bashrc

cat > ${HOME}/.psqlrc <<-\EOF
\set PROMPT1 '%`date +%H:%M:%S` (%n@%M:%>)%/%R%#%x '
\set PROMPT2 '%M %n@%/%R%# '
\timing on
EOF
Enter fullscreen mode Exit fullscreen mode

3、Install graph-node

Tutorial:

3.1 Create log files directory

USER="`whoami`"
sudo mkdir -p /var/log/graph
sudo chown -R ${USER}:${USER} /var/log/graph
Enter fullscreen mode Exit fullscreen mode

3.2 Create config files directory

USER="`whoami`"
sudo mkdir -p /etc/graph-node
sudo chown -R ${USER}:${USER} /etc/graph-node
touch /etc/graph-node/expensive-queries.txt
Enter fullscreen mode Exit fullscreen mode

3.3 Install Rust

安装或者更新:

cargo -V
if [ "$?" == 0 ] ; then
    rustup update stable
else
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    source ${HOME}/.cargo/env
fi
cargo -V
Enter fullscreen mode Exit fullscreen mode

3.4 Download graph-node complier,we have modified the graphnode code! Make sure you download it from our github!

sudo apt install -y cmake

cd ${HOME}
git clone https://github.com/gzliudan/graph-node graph-node.xdc
cd graph-node.xdc
git checkout -b xdc origin/xdc
cargo build --release
Enter fullscreen mode Exit fullscreen mode

3.5 Create a startup Script

cat > ${HOME}/start-graph-indexer.sh <<-\EOF
#!/bin/bash

export CHAIN="$1"

# export GRAPH_LOG="trace"
# export GRAPH_LOG="debug"
export GRAPH_LOG_TIME_FORMAT="%Y-%m-%d %H:%M:%S"
LOG_FILE="/var/log/graph/indexer-${CHAIN}-`/usr/bin/date +%Y%m%d`.log"

${HOME}/graph-node.xdc/target/release/graph-node \
  --node-id indexer_${CHAIN} \
  --postgres-url postgresql://graph:daniel2022@localhost:5432/${CHAIN}_db \
  --ethereum-rpc ${CHAIN}:https://arpc.${CHAIN}.network \
  --ipfs 127.0.0.1:5001 \
  --expensive-queries-filename /etc/graph-node/expensive-queries.txt \
  >> ${LOG_FILE} 2>&1
EOF
Enter fullscreen mode Exit fullscreen mode

chmod +x ${HOME}/start-graph-indexer.sh

3.6 Create service files

USER="`whoami`"
sudo tee /usr/lib/systemd/system/graph-indexer@.service <<EOF
[Unit]
Description=Graph indexer node %i
After=network.target ipfs.service
Wants=network.target ipfs.service

[Service]
User=${USER}
Group=${USER}
WorkingDirectory=/home/${USER}/
StandardOutput=journal
StandardError=journal
Type=simple
Restart=always
RestartSec=5
ExecStart=/home/${USER}/start-graph-indexer.sh %i

[Install]
WantedBy=default.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable graph-indexer@apothem
# sudo systemctl disable graph-indexer@apothem
sudo systemctl start graph-indexer@apothem
systemctl status graph-indexer@apothem
# sudo systemctl stop graph-indexer@apothem
Enter fullscreen mode Exit fullscreen mode

3.7 Time setting to restart the indexer service

examine cron job: sudo crontab -l -u root
add corn job to root account: sudo crontab -e -u root

restart service daily: 0 0 * * * /usr/bin/systemctl restart graph-indexer@apothem.service

examine the task: sudo crontab -l -u root

3.8 TODO

  • you can always pull the source code, comply and restart the service by adding a timed task if you need.

Query with sample

Tutorial: https://thegraph.com/hosted-service/subgraph/gzliudan/bad-token-mumbai

http://103.101.129.136:8000/subgraphs/name/gzliudan/bad-token-subgraph-apothem

{
  erc20Contracts(first: 5) {
    id
    name
    symbol
    decimals
    totalSupply {
      value
      valueExact
    }
  }
  accounts(first: 5) {
    id
    isErc20
    blackLists {
      id
    }
    Erc20balances {
      id
    }
  }
  blackLists(first: 5) {
    id
    members {
      id
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Discussion (14)

Collapse
ncode profile image
ncode

Can we do this using docker?
like with a simple docker-compose up

Collapse
pro100skm profile image
Mr. Popovich

totally agree. by making it with docker-compose we will have only one config file and only one line of code to run it

Collapse
mrblockchain22 profile image
Salomon Morales

I agree with the approach of Docker but we have to keep in mind that if Docker version gets updated, to ensure that whatever it's deploy did not get affected by the vers update. That's my only concern but I agree that we should simplify what we can and make it easier (ready-to-deploy).

Thread Thread
pro100skm profile image
Mr. Popovich

Maybe we talk about different things. What im saying is to make one docker-compose file that will have all scripts. No need to deploy image to docker-hub.

Collapse
walterblueu profile image
Jon McBee

I am in favor in taking whatever approach simplifies usage of the tool for the developer community as it will not only make this easier to consume but easier to support when issues arise.

Collapse
eoaks profile image
Eduardo Robles

I've been trying to implement a subgraph using graph-protocol's graph-node by changing this file to have ethereum: 'mainnet:https://rpc.apothem.network'.

But running docker-compose up with this gives the following error.
Aug 29 20:46:40.700 WARN Trying again after eth_getBlockByNumber(0, false) RPC call failed (attempt #11) with result Err(Decoder error: Error("invalid length 43, expected a (both 0x-prefixed or not) hex string with length of 40", line: 0, column: 0)), provider: mainnet-rpc-0

I was kindly pointed to this post which does say that it is possible to have a subgraph, but haven't been able to tell the difference from my code to the one in the examples.

Can someone point me in the right direction here? What am i missing?

Collapse
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author

you need to instal from the adress we provide

Collapse
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author

looking into it

Collapse
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author

Hi
You need to use the GitHub we provide we have modified the graph source code to be adoptable with xdc network.

Collapse
eoaks profile image
Eduardo Robles

I have cloned gzliudan/graph-node but still get these errors | Aug 31 13:12:23.552 WARN Trying again after eth_getBlockByNumber(0, false) RPC call failed (attempt #16) with result Err(Decoder error: Error("invalid length 43, expected a (both 0x-prefixed or not) hex string with length of 40", line: 0, column: 0)), provider: mainnet-rpc-0 and when trying to deploy Failed to deploy to Graph node http://localhost:8020/: network not supported by registrar: no network mainnet found on chain ethereum

Thread Thread
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author

let me check

Thread Thread
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author

you havent switched to the xdc branch

git checkout -b xdc origin/xdc

Collapse
belle profile image
Belle

As what Mr. Popovich has said, it will be good to have a ready made product that everyone can use through docker-compose. The solution provided is great but then in order for it to work, you have make it from scratch by going through all the steps and there is no guarantee of its outcome. The room for error is huge compared to having it in docker-compose.

Collapse
xu_zhaolin_fcf881856ae0b5 profile image
xu zhaolin Author • Edited on

Technically speaking, it can be done with docker. But the current version is unstable,we dont suggest to use docker. In addition, in the production enviroment, ipfs and pg database is privately deployed in a cluster manner rather than putting them in docker-compose.